Blog

n a recent project, I needed to do some time-based imputation across a large set of data. I tried to implement my own solution with moderate success before scouring the internet for a solution. After an hour or so, I came across this article about the Spark-TS package.

When coming to Spark from a background in R or Python Pandas, you’ll likely get tripped up on a few things. The most notable of these is the difference between R and Python dataframe apis and the Spark dataframe API. Furthermore, not all models in Spark are fit with a dataframe and the inter loop between dataframes and RDD (Resilient distributed datasets) are not so obvious.