Deduplicate and Aggregate User Actions with Latest Session

PYSPARK coding challenge · Difficulty: hard · Topic: DataFrame Deduplication · +200 XP

Problem

user_actions records one row per action, tagged with the session it happened in. A user can have several sessions, and a session can contain several actions.

For each user, report the timestamp of their most recent action and how many DISTINCT sessions they have.

Example Input - `user_actions`

| user_id | session_id | action_time | action_type |
| --- | --- | --- | --- |
| 101 | S1 | 2024-01-01 10:00:00 | click |
| 101 | S1 | 2024-01-01 10:05:00 | scroll |
| 101 | S2 | 2024-01-01 11:00:00 | click |
| 102 | S3 | 2024-01-02 09:00:00 | click |
| 102 | S3 | 2024-01-02 09:30:00 | scroll |
| 103 | S4 | 2024-01-03 08:00:00 | click |
| 103 | S5 | 2024-01-03 09:00:00 | scroll |
| 103 | S5 | 2024-01-03 09:15:00 | click |

Expected Output

| user_id | latest_session_time | unique_session_count |
| --- | --- | --- |
| 101 | 2024-01-01 11:00:00 | 2 |
| 102 | 2024-01-02 09:30:00 | 1 |
| 103 | 2024-01-03 09:15:00 | 2 |

Explanation

Count sessions, not rows. User 102 has two actions but both are in session S3, so their count is 1 -- a plain count() would say 2. User 101 acted three times across S1 and S2, giving 2, and user 103 acted three times across S4 and S5, also 2.

latest_session_time is the maximum action_time for the user, regardless of which session it belongs to: user 102's latest is 09:30 from the second S3 row.

Notes

• The DataFrame is created for you - do not recreate it
• Build a DataFrame called df_result and finish with df_result.show()
• Return user_id, latest_session_time, unique_session_count

What this PYSPARK challenge teaches you

“Deduplicate and Aggregate User Actions with Latest Session” is a hard-level PYSPARK challenge focused on DataFrame Deduplication. Working through it gives you hands-on practice with dropDuplicates, window, groupBy — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.

Concepts covered

• dropDuplicates
• window
• groupBy

How to approach it

If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:

1. Your result should return: user_id, latest_session_time,.
2. Compare your output to the Expected Output — the columns, values and row order must match exactly.

Where this comes up

Variations of this problem have been reported in interviews at Meta, Twitter, LinkedIn. Interviewers use it to check whether you can express the logic cleanly and reason about correctness on edge cases such as ties, nulls and empty groups.

How to practise it on PySpark.in

Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.

Related PYSPARK challenges

• Fix the Broken Pipeline
• Optimize the 100M-Row Join
• Parse Apache Logs with Regex
• Optimize Small DataFrame Join with Broadcast
• Optimize Average Rating Calculation for Products
• Pivot Sales Data to Show Monthly Totals by Product

Frequently asked questions

Do I need to install Spark or a database to solve this?

No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.

Is this challenge free?

Yes - the problem, the sample dataset, the hints and unlimited test runs are free.

What level is it?

It is rated hard and covers DataFrame Deduplication.

Solve this challenge free on PySpark.in