Testing Spark Dataframes
Testing Spark Dataframe transforms is essential and can be accomplished in a more reusable manner. The way, I generally accomplish that is to
- Read the expected and test Dataframe, and
- Invoke the desired transform, and
- Calculate the difference between dataframes. The only caveat in calculating the difference is that in …