Parse Apache Logs with Regex
PYSPARK coding challenge · Difficulty: medium · Topic: String Processing · +120 XP
Problem
Each row of log_df holds one raw Apache access-log line in a column called value. Pull the five fields out of it with a single regular expression.
A line looks like this:
`
192.168.1.1 - - [01/Jan/2024] "GET /home HTTP/1.1" 200 1024
`
Extract the client IP, the HTTP method, the requested path, the status code and the byte count. Status and bytes must come out as integers, not strings.
Schema - `log_df`
| Column | Type | | --- | --- | | value | string (one raw log line) |
Example Input - `log_df`
| value | | --- | | 192.168.1.1 - - [01/Jan/2024] "GET /home HTTP/1.1" 200 1024 | | 10.0.0.5 - - [01/Jan/2024] "POST /login HTTP/1.1" 401 512 | | 192.168.1.2 - - [01/Jan/2024] "GET /about HTTP/1.1" 200 768 |
Expected Output
| ip | method | path | status | bytes | | --- | --- | --- | --- | --- | | 192.168.1.1 | GET | /home | 200 | 1024 | | 10.0.0.5 | POST | /login | 401 | 512 | | 192.168.1.2 | GET | /about | 200 | 768 |
Explanation
Every line is parsed and every line is returned - there is no filtering in this task. The 401 row stays in the output alongside the two 200s.
One regex with five capture groups does the whole job, and regexp_extract(col, pattern, n) pulls out group n:
- group 1
(\S+)- the IP, the first run of non-space characters - groups 2 and 3 - the method and path, taken from inside the quoted request
- groups 4 and 5
(\d+)- status and bytes, the two numbers at the end
regexp_extract always returns a string, so status and bytes need an explicit .cast("int"). Without it they still print the right digits but the column is a string, and the comparison fails on type.
Notes
- The DataFrame is created for you - do not recreate it
- Build a DataFrame called
df_resultand finish withdf_result.show() - Return all rows:
ip,method,path,status(int),bytes(int)
What this PYSPARK challenge teaches you
“Parse Apache Logs with Regex” is a medium-level PYSPARK challenge focused on String Processing. Working through it gives you hands-on practice with regexp_extract, regex, parsing — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.
Concepts covered
- regexp_extract
- regex
- parsing
How to approach it
If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:
- Compare your output to the Expected Output — the columns, values and row order must match exactly.
Where this comes up
Variations of this problem have been reported in interviews at Netflix, Cloudflare, Uber. Interviewers use it to check whether you can express the logic cleanly and reason about correctness on edge cases such as ties, nulls and empty groups.
How to practise it on PySpark.in
Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.
Related PYSPARK challenges
- Fix the Broken Pipeline
- Optimize the 100M-Row Join
- Optimize Small DataFrame Join with Broadcast
- Optimize Average Rating Calculation for Products
- Deduplicate and Aggregate User Actions with Latest Session
- Pivot Sales Data to Show Monthly Totals by Product
Frequently asked questions
Do I need to install Spark or a database to solve this?
No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.
Is this challenge free?
Yes - the problem, the sample dataset, the hints and unlimited test runs are free.
What level is it?
It is rated medium and covers String Processing.