Fix the Broken Pipeline

PYSPARK coding challenge · Difficulty: medium · Topic: DataFrame Operations · +75 XP

Problem

Your ETL pipeline joins df_orders with df_customers. A bug in the customer feed means customer_id = 1 appears twice (Alice and Alice_dup), so every order for that customer comes back duplicated and inflates revenue on the dashboards.

Fix the join so each order appears exactly once, keeping one row per customer from the lookup.

Three input orders must produce three output rows.

Example Input - `df_orders`

| order_id | customer_id | amount |
| --- | --- | --- |
| 101 | 1 | 250 |
| 102 | 2 | 180 |
| 103 | 1 | 90 |

Example Input - `df_customers`

| customer_id | name |
| --- | --- |
| 1 | Alice |
| 1 | Alice_dup |
| 2 | Bob |

Expected Output

| customer_id | order_id | amount | name |
| --- | --- | --- | --- |
| 1 | 101 | 250 | Alice |
| 2 | 102 | 180 | Bob |
| 1 | 103 | 90 | Alice |

Notes

• The DataFrame is created for you - do not recreate it
• Build a DataFrame called df_result and finish with df_result.show()

What this PYSPARK challenge teaches you

“Fix the Broken Pipeline” is a medium-level PYSPARK challenge focused on DataFrame Operations. Working through it gives you hands-on practice with joins, debugging, etl — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.

Concepts covered

• joins
• debugging
• etl

How to approach it

If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:

1. This is a DataFrame Operations problem — review the matching pyspark concept.
2. Compare your output to the Expected Output — the columns, values and row order must match exactly.

Where this comes up

Variations of this problem have been reported in interviews at Uber, Databricks, Netflix. Interviewers use it to check whether you can express the logic cleanly and reason about correctness on edge cases such as ties, nulls and empty groups.

How to practise it on PySpark.in

Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.

Related PYSPARK challenges

• Top 3 Customers by Total Sales (PySpark)
• Orders per Customer per Month (PySpark)
• Customers With More Than 5 Orders (PySpark)
• Total Sales by Product Category (PySpark Join)
• Optimize the 100M-Row Join
• Parse Apache Logs with Regex

Frequently asked questions

Do I need to install Spark or a database to solve this?

No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.

Is this challenge free?

Yes - the problem, the sample dataset, the hints and unlimited test runs are free.

What level is it?

It is rated medium and covers DataFrame Operations.

Solve this challenge free on PySpark.in