Clean Order Prices by Handling Nulls and Filtering Negatives
PYSPARK coding challenge · Difficulty: medium · Topic: Null Handling and Data Cleaning · +100 XP
The orders DataFrame has an order_id, a price and a discount; either amount can be NULL. Produce a cleaned DataFrame df_result:
1. treat a NULL price as 0 and a NULL discount as 0;
2. compute net_price = price - discount;
3. keep only rows where net_price is strictly greater than 0.
Return columns order_id and net_price, ordered by order_id, and call df_result.show(). Do not drop a row just because one of its amounts is NULL -- fill it first, then decide.
Example Input
`
orders (already defined -- do not redefine it):
order_id | price | discount
1 | 100.0 | 10.0
2 | NULL | 5.0
3 | 50.0 | NULL
4 | 20.0 | 20.0
5 | 30.0 | 45.0
6 | 80.0 | 5.0
7 | NULL | NULL
8 | 12.5 | 0.25
`
Expected Output
`
+--------+---------+
|order_id|net_price|
+--------+---------+
| 1| 90.0| | 3| 50.0| | 6| 75.0| | 8| 12.25|
+--------+---------+
`
Order 4 nets exactly 0 and order 7 is all NULL (0 - 0), so neither is kept.
What this PYSPARK challenge teaches you
“Clean Order Prices by Handling Nulls and Filtering Negatives” is a medium-level PYSPARK challenge focused on Null Handling and Data Cleaning. Working through it gives you hands-on practice with na.fill, withColumn, filter, data cleaning — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.
Concepts covered
- na.fill
- withColumn
- filter
- data cleaning
How to approach it
If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:
- Fill first: orders.na.fill({"price": 0.0, "discount": 0.0}) -- dropping NULL rows would lose order 3.
- Then withColumn("net_price", col("price") - col("discount")).
- The filter is strict: net_price > 0, so a net of exactly 0 is removed.
- Select only order_id and net_price, ordered by order_id.
How to practise it on PySpark.in
Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.
Related PYSPARK challenges
- Fix the Broken Pipeline
- Optimize the 100M-Row Join
- Parse Apache Logs with Regex
- Optimize Small DataFrame Join with Broadcast
- Optimize Average Rating Calculation for Products
- Deduplicate and Aggregate User Actions with Latest Session
Helpful resources
Frequently asked questions
Do I need to install Spark or a database to solve this?
No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.
Is this challenge free?
Yes - the problem, the sample dataset, the hints and unlimited test runs are free.
What level is it?
It is rated medium and covers Null Handling and Data Cleaning.