Optimize Small DataFrame Join with Broadcast

PYSPARK coding challenge · Difficulty: easy · Topic: Broadcast Join Optimisation · +50 XP

Problem

Each customer row carries a short region_code. The reporting team wants the

region's full name beside every customer instead of the code.

regions is a tiny lookup -- three rows -- and customers is the large side.

Broadcasting the small side sends a copy of it to every executor, so Spark can

join without shuffling the large DataFrame across the cluster.

Example Input - `customers`

| customer_id | region_code |
| --- | --- |
| 1001 | NA |
| 1002 | EU |
| 1003 | NA |
| 1004 | APAC |
| 1005 | EU |
| 1006 | NA |

Example Input - `regions`

| region_code | region_name |
| --- | --- |
| NA | North America |
| EU | Europe |
| APAC | Asia Pacific |

Expected Output

| customer_id | region_name |
| --- | --- |
| 1001 | North America |
| 1002 | Europe |
| 1003 | North America |
| 1004 | Asia Pacific |
| 1005 | Europe |
| 1006 | North America |

Notes

Both DataFrames are created for you - do not recreate them
Build a DataFrame called df_result and finish with df_result.show()

Success Criteria

Your DataFrame has two columns: customer_id and region_name.
Every customer appears exactly once, with its region's full name.
Rows come back ordered by customer_id, and you finish by calling .show().
The grader compares the rows you produce, so an ordinary join also passes --

but broadcasting the small side is the point of the exercise, and the thing

that keeps the large DataFrame from being shuffled.

What this PYSPARK challenge teaches you

“Optimize Small DataFrame Join with Broadcast” is a easy-level PYSPARK challenge focused on Broadcast Join Optimisation. Working through it gives you hands-on practice with broadcast, join, optimization — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.

Concepts covered

broadcast
join
optimization

How to approach it

If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:

`broadcast()` marks the SMALL side of a join, so Spark ships a copy of it to every executor instead of shuffling the large side across the cluster. Import it with `from pyspark.sql.functions import broadcast`.
Both DataFrames share the `region_code` column, so one key as a string is enough and no column ends up duplicated: `customers.join(broadcast(regions), 'region_code')`.
Keep only what the output needs -- `.select('customer_id', 'region_name')` -- then `.orderBy('customer_id')` and finish with `.show()`.

Where this comes up

Variations of this problem have been reported in interviews at Databricks, Spotify, Twitter. Interviewers use it to check whether you can express the logic cleanly and reason about correctness on edge cases such as ties, nulls and empty groups.

How to practise it on PySpark.in

Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.

Related PYSPARK challenges

Helpful resources

Frequently asked questions

Do I need to install Spark or a database to solve this?

No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.

Is this challenge free?

Yes - the problem, the sample dataset, the hints and unlimited test runs are free.

What level is it?

It is rated easy and covers Broadcast Join Optimisation.

Solve this challenge free on PySpark.in