Testing Spark Dataframes

Testing Spark Dataframe transforms is essential and can be accomplished in a more reusable manner. The way, I generally accomplish that is to

• Read the expected and test Dataframe, and
• Invoke the desired transform, and
• Calculate the difference between dataframes. The only caveat in calculating the difference is that in …

in spark · Wed 17 January 2018

Parallel Orchestration of Spark ETL Processing

I have been working a lot on Spark and Scala. I have really like scala as a language, due to its numerous advantages over Java, the foremost being that for a simpler API having Type classes and Default Method Arguments does wonders. Also, idiomatic scala code uses higher order functions …

in spark · Sat 28 October 2017