Working with Skewed Data: The Iterative Broadcast - Rob Keevil & Fokko Driesprong
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about Spark jobs stuck at the last task or hitting skewed joins, and when asking how to implement the iterative broadcast join.
Recommended by 3 people across 3 different months, 2017 to 2020.
What it is
A Spark Summit Europe 2017 conference talk by Rob Keevil and Fokko Driesprong on joining skewed datasets in Spark. It explains why skew overloads a few nodes, why broadcast joins fall short for large tables, and demonstrates the iterative broadcast join technique developed on ING Bank transaction data.
What people said
Recommends Fokko Driesprong · 2017-11 · Stack Overflow · on “Spark job getting stuck at 99% and doesn't continue”
Because of the join operations, all records with the same key are shuffled to the same executor. If you data is skewed, which means that there is one or a few keys which are very dominant in terms of the number of rows. Then this single executor which has to process all these rows. Essentially your Spark job becomes single threaded since this single key needs to be processed by a single thread. Repartitioning will not help since your join operation will shuffle the data again by hashing the join key. You could try to increase the number of partitions in case of an unlucky hash. This video explains the problems, and suggests a solution: [this video] Cheers, Fokko
Recommends jaspreet kaur bassan · 2018-01 · Stack Overflow · on “Pyspark job being stuck at the final task”
The operation was being stuck at the final task because of the skewed data being joined to a huge dataset. The key that was joining the two dataframes was heavily skewed. The problem was solved for now by removing the skewed data from the dataframe. If you must include the skewed data, you can use iterative broadcast joins ( [LINK] ). Look into this informative video for more details [this video]
Recommends Ram Ghadiyaram · 2019-05 · Stack Overflow · on “Broadcast Hash Join (BHJ) in Spark for full outer join (outer, full, fullouter)”
iterative-broadcast-join [LINK] this project talks about that with complete example just check. this is the video of that project iterative-broadcast-join [LINK] this project talks about that with complete example just check. this is the video [this video] of that project
Cites Fokko Driesprong · 2017-11 · Stack Overflow · on “How to join big dataframes in Spark SQL? (best practices, stability, performance)”
That are a lot of questions. Allow me to answer these one by one: Your number of executors is most of the time variable in a production environment. This depends on the available resources. The number of partitions is important when you are performing shuffles. Assuming that your data is now skewed, you can lower the load per task by increasing the number of partitions. A task should ideally take a couple of minus. If the task takes too long, it is possible that your container gets pre-empted and the work is lost. If the task takes only a few milliseconds, the overhead of starting the task gets dominant. The level of parallelism and tuning your executor sizes, I would like to refer to the ex
Cites marie20 · 2020-09 · Stack Overflow · on “Iterative Broadcast Join in Spark SQL”
I recently came across this talk [this video] about dealing with Skew in Spark SQL by using " Iterative " Broadcast Joins to improve query performance when joining a large table with another not so small table. The talk advises to tackle such scenarios using "Iterative Broadcast Joins". Unfortunately, the talk doesn't probe deep enough for me to understand its implementation. Hence, I was hoping if someone could please shed some light on how to implement this Iterative Broadcast Join in Spark SQL with few examples. How do I implement the same using Spark SQL queries with the SQL-API ? Note: I am using Spark SQL query 2.4 Any help is appreciated. Thanks