Generate Missing IDs for Fruits
PYSPARK coding challenge · Difficulty: medium · Topic: Window Functions · +100 XP
Fruits records a fruit on every row, but the id is sometimes missing.
Tables
| column | type | description | | --- | --- | --- | | id | INT | the id, which may be NULL | | fruit | VARCHAR | the fruit name |
Every fruit has at least one row carrying a non-NULL id, and all non-NULL
ids for the same fruit are identical.
Add a column new_id holding that fruit's known id prefixed with 00, on
every row of the fruit — the rows where id is NULL included. Leave id
and fruit exactly as they are.
Sort by fruit ascending, then by id ascending with NULL ids last.
Example input
Fruits
| id | fruit | | --- | --- | | 1 | apple | | NULL | apple | | 2 | banana | | 2 | banana | | NULL | banana |
Expected output
| id | fruit | new_id | | --- | --- | --- | | 1 | apple | 001 | | NULL | apple | 001 | | 2 | banana | 002 | | 2 | banana | 002 | | NULL | banana | 002 |
apple's known id is 1, so every apple row gets 001 — including the NULL
one. banana's known id is 2, so all three banana rows get 002. The id
column itself keeps its NULLs.
What this PYSPARK challenge teaches you
“Generate Missing IDs for Fruits” is a medium-level PYSPARK challenge focused on Window Functions. Working through it gives you hands-on practice with window-functions, null-handling, string-functions — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.
Concepts covered
- window-functions
- null-handling
- string-functions
How to approach it
If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:
- A window function can see the whole fruit from any of its rows: MAX(id) OVER (PARTITION BY fruit) hands every row that fruit's id.
- MAX and MIN skip NULLs, so on a partition holding {2, 2, NULL} either one returns 2 -- no COALESCE needed to find the value.
- The prefix is string concatenation, not arithmetic: CONCAT('00', 1) is '001'. In PySpark cast the number to string first. And in MySQL, ORDER BY id puts NULLs FIRST -- 'id IS NULL' as a preceding sort key pushes them last.
Where this comes up
Variations of this problem have been reported in interviews at Adobe. Interviewers use it to check whether you can express the logic cleanly and reason about correctness on edge cases such as ties, nulls and empty groups.
How to practise it on PySpark.in
Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.
Related PYSPARK challenges
- Top 3 Products per Category
- Running Total Revenue
- Median Salary per Department
- Latest Order Per Customer
- 3-Day Rolling Sum of Sales
- 7-Day Rolling Purchase Amount by Customer
Helpful resources
Frequently asked questions
Do I need to install Spark or a database to solve this?
No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.
Is this challenge free?
Yes - the problem, the sample dataset, the hints and unlimited test runs are free.
What level is it?
It is rated medium and covers Window Functions.