Optimize Join Operation Using Broadcast in AQE

PYSPARK coding challenge · Difficulty: hard · +200 XP

You are given two DataFrames: customers and orders. The orders DataFrame is significantly larger than the customers DataFrame. Your task is to join these two DataFrames on the customer_id column using Spark's Adaptive Query Execution (AQE) to efficiently handle the skewed join operation. Use broadcasting for the smaller DataFrame to optimize the query.

Example input

`

customers DataFrame:

customer_id | name

1 | Alice

2 | Bob

3 | Cathy

orders DataFrame:

order_id | customer_id | amount

101 | 1 | 250

102 | 1 | 300

103 | 2 | 150

104 | 3 | 400

105 | 3 | 500

106 | 3 | 450

`

Expected output

`

+--------+-----------+------+-----+

|order_id|customer_id|amount|name |

+--------+-----------+------+-----+

|     101|          1|   250|Alice|
|     102|          1|   300|Alice|
|     103|          2|   150|Bob  |
|     104|          3|   400|Cathy|
|     105|          3|   500|Cathy|
|     106|          3|   450|Cathy|

+--------+-----------+------+-----+

`