Time Between Consecutive Vehicle Events

PYSPARK coding challenge · Difficulty: medium · Topic: Window Functions · +150 XP

Problem

You are analysing vehicle movement through an Uber toll plaza. The toll_log DataFrame holds every crossing, and a vehicle may cross many times in a day.

For each crossing, work out how long it had been since that same vehicle's previous crossing.

The previous crossing is found per vehicle, in toll_time order -- never by the order rows happen to sit in the DataFrame.

Input DataFrame — `toll_log`

| Column | Type |
| --- | --- |
| toll_id | INT |
| vehicle_id | STRING |
| toll_time | TIMESTAMP |

Rules

previous_toll_time is that vehicle's immediately preceding crossing. Other vehicles are irrelevant.
hours_since_previous is the elapsed time in hours, rounded to 2 decimal places -- so 1 hour 25 minutes is 1.42, not 1 and not 1.4. (Numbers are compared by value, so 2.5 and 2.50 are both accepted.)
For a vehicle's first crossing, both previous_toll_time and hours_since_previous are NULL.
Every input row appears in the output. Nothing is filtered out.

Example Input

| toll_id | vehicle_id | toll_time |
| --- | --- | --- |
| 1 | V101 | 2024-01-15 08:00:00 |
| 2 | V101 | 2024-01-15 10:30:00 |
| 3 | V101 | 2024-01-15 15:00:00 |
| 4 | V102 | 2024-01-15 09:00:00 |
| 5 | V102 | 2024-01-15 13:00:00 |
| 6 | V103 | 2024-01-15 07:15:00 |
| 7 | V103 | 2024-01-15 08:45:00 |
| 8 | V103 | 2024-01-15 12:15:00 |
| 9 | V104 | 2024-01-15 11:00:00 |
| 10 | V105 | 2024-01-15 20:20:00 |
| 11 | V105 | 2024-01-15 17:35:00 |
| 12 | V105 | 2024-01-15 19:00:00 |

Expected Output

| vehicle_id | toll_time | previous_toll_time | hours_since_previous |
| --- | --- | --- | --- |
| V101 | 2024-01-15 08:00:00 | NULL | NULL |
| V101 | 2024-01-15 10:30:00 | 2024-01-15 08:00:00 | 2.50 |
| V101 | 2024-01-15 15:00:00 | 2024-01-15 10:30:00 | 4.50 |
| V102 | 2024-01-15 09:00:00 | NULL | NULL |
| V102 | 2024-01-15 13:00:00 | 2024-01-15 09:00:00 | 4.00 |
| V103 | 2024-01-15 07:15:00 | NULL | NULL |
| V103 | 2024-01-15 08:45:00 | 2024-01-15 07:15:00 | 1.50 |
| V103 | 2024-01-15 12:15:00 | 2024-01-15 08:45:00 | 3.50 |
| V104 | 2024-01-15 11:00:00 | NULL | NULL |
| V105 | 2024-01-15 17:35:00 | NULL | NULL |
| V105 | 2024-01-15 19:00:00 | 2024-01-15 17:35:00 | 1.42 |
| V105 | 2024-01-15 20:20:00 | 2024-01-15 19:00:00 | 1.33 |

Explanation

V101 crosses three times: the first has nothing before it, then 2h30m

(2.50) and 4h30m (4.50).

V103 shows why minutes matter -- 07:15 to 08:45 is 1.50, not 1.

V104 crosses once all day. Its row still appears, with NULL in both

columns.

V105 is the one to watch. Its three crossings are stored out of order --

toll_id 10 is 20:20, 11 is 17:35, 12 is 19:00. Ordering the window by

toll_time puts them right: 17:35 first, then 19:00 (1h25m -> 1.42), then

20:20 (1h20m -> 1.33). Anything that leans on toll_id, or on the order rows

happen to sit in, produces a different sequence entirely.

Notes

The DataFrame is created for you -- do not recreate it
Build a DataFrame called df_result and finish with df_result.show()
Return exactly: vehicle_id, toll_time, previous_toll_time, hours_since_previous
Functions to use: Window.partitionBy(...).orderBy(...), F.lag, F.unix_timestamp

What this PYSPARK challenge teaches you

“Time Between Consecutive Vehicle Events” is a medium-level PYSPARK challenge focused on Window Functions. Working through it gives you hands-on practice with lag, Window, partitionBy, unix_timestamp, decimal cast, NULL handling — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.

Concepts covered

lag
Window
partitionBy
unix_timestamp
decimal cast
NULL handling

How to approach it

If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:

Each row needs to see the row before it for the same vehicle. That is exactly what F.lag() over a window does -- no self-join, no groupBy.
Define the window first: w = Window.partitionBy('vehicle_id').orderBy('toll_time'). partitionBy keeps vehicles apart; orderBy is what makes 'previous' mean previous in time.
Subtracting two timestamp columns does not give hours. Convert both with F.unix_timestamp(...), subtract to get seconds, then divide by 3600.
Either F.round(x, 2) or a .cast('decimal(10,2)') works. round leaves a double, which prints 2.5; the decimal cast has a fixed scale and prints 2.50. Values are compared numerically, so both are accepted -- what matters is that the number is right to 2 decimals.
The first crossing of each vehicle has no previous row, so lag() is NULL. NULL flows through the subtraction and the division on its own, so hours_since_previous ends up NULL with no special handling. Do not filter those rows out.

Where this comes up

Variations of this problem have been reported in interviews at Uber. Interviewers use it to check whether you can express the logic cleanly and reason about correctness on edge cases such as ties, nulls and empty groups.

How to practise it on PySpark.in

Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.

Related PYSPARK challenges

Frequently asked questions

Do I need to install Spark or a database to solve this?

No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.

Is this challenge free?

Yes - the problem, the sample dataset, the hints and unlimited test runs are free.

What level is it?

It is rated medium and covers Window Functions.

Solve this challenge free on PySpark.in