Log Analysis Dashboard

Spark tutorial · PySpark.in

Python + PySpark Hands-On Example

Project Overview

Parse raw Apache web server logs, transform them into a structured Spark DataFrame, and extract actionable insights — all in one runnable script.

Project Purpose

Web servers generate thousands of log entries every minute. This project teaches you how to use PySpark's distributed engine to process those logs efficiently — a skill that scales from a few KB of sample data to terabytes in production.

You will learn how to: parse unstructured text with regex, build a typed Spark DataFrame, and run aggregations to answer real operational questions like "which endpoints are slowest?" or "is our error rate acceptable?".

Key Skills Covered

Dataset — Apache Log Format

Embedded directly in the code — no file upload needed

Each line follows the Combined Log Format used by Apache and Nginx:

IP - - [timestamp] "METHOD endpoint HTTP/1.1" status_code bytes

Sample Log Records

192.168.1.1 - - [01/Jan/2024:10:00:01] "GET /home HTTP/1.1" 200 1024

192.168.1.2 - - [01/Jan/2024:10:00:02] "POST /login HTTP/1.1" 401 512

10.0.0.5 - - [01/Jan/2024:10:00:03] "GET /products HTTP/1.1" 200 2048

10.0.0.7 - - [01/Jan/2024:10:00:05] "DELETE /user/42 HTTP/1.1" 403 256

192.168.1.1 - - [01/Jan/2024:10:00:09] "GET /dashboard HTTP/1.1" 500 0

192.168.1.2 - - [01/Jan/2024:10:00:14] "POST /api/submit HTTP/1.1" 500 0

How It Works — Step by Step

Step 1: Parse Raw Logs

Use a regex pattern to extract IP address, timestamp, HTTP method, endpoint, status code, and bytes from each raw log line into structured columns.

Step 2: Build a Spark DataFrame

Convert the parsed RDD into a typed Spark DataFrame with an explicit schema. This enables SQL-style querying and distributed processing.

Step 3: Status Code Distribution

Group by status_code and count occurrences to see how many 200 OK, 404 Not Found, 500 Server Error, etc. requests your server received.

Step 4: Error Rate Analysis

Filter rows where status_code >= 400, count them, and calculate the error percentage. High error rates indicate server or client problems.

Step 5: Top IPs

Identify the IP addresses making the most requests. This can reveal heavy users, bots, or potential DDoS sources.

Step 6: Popular Endpoints

Rank your URLs by visit count to understand which pages or API routes get the most traffic.

Step 7: Data Transferred

Sum the bytes column across all requests to measure total bandwidth consumed — useful for capacity planning.

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col, count, desc, sum as spark_sum

import re

# Initialize Spark Session

spark = SparkSession.builder \

    .appName("Log Analysis Dashboard") \

    .master("local[*]") \

    .getOrCreate()

spark.sparkContext.setLogLevel("ERROR")

# ============================================================

# DATASET: Sample Apache Web Server Logs (embedded)

# ============================================================

log_data = [

    '192.168.1.1 - - [01/Jan/2024:10:00:01 +0000] "GET /home HTTP/1.1" 200 1024',

    '192.168.1.2 - - [01/Jan/2024:10:00:02 +0000] "POST /login HTTP/1.1" 401 512',

    '10.0.0.5  - - [01/Jan/2024:10:00:03 +0000] "GET /products HTTP/1.1" 200 2048',

    '192.168.1.1 - - [01/Jan/2024:10:00:04 +0000] "GET /about HTTP/1.1" 200 768',

    '10.0.0.7  - - [01/Jan/2024:10:00:05 +0000] "DELETE /user/42 HTTP/1.1" 403 256',

    '192.168.1.3 - - [01/Jan/2024:10:00:06 +0000] "GET /home HTTP/1.1" 200 1024',

    '10.0.0.5  - - [01/Jan/2024:10:00:07 +0000] "GET /notfound HTTP/1.1" 404 128',

    '192.168.1.2 - - [01/Jan/2024:10:00:08 +0000] "POST /login HTTP/1.1" 200 512',

    '192.168.1.1 - - [01/Jan/2024:10:00:09 +0000] "GET /dashboard HTTP/1.1" 500 0',

    '10.0.0.9  - - [01/Jan/2024:10:00:10 +0000] "GET /home HTTP/1.1" 200 1024',

    '192.168.1.4 - - [01/Jan/2024:10:00:11 +0000] "PUT /user/10 HTTP/1.1" 200 300',

    '192.168.1.1 - - [01/Jan/2024:10:00:12 +0000] "GET /api/data HTTP/1.1" 200 4096',

    '10.0.0.5  - - [01/Jan/2024:10:00:13 +0000] "GET /home HTTP/1.1" 200 1024',

    '192.168.1.2 - - [01/Jan/2024:10:00:14 +0000] "POST /api/submit HTTP/1.1" 500 0',

    '10.0.0.7  - - [01/Jan/2024:10:00:15 +0000] "GET /products HTTP/1.1" 200 2048',

]

# ============================================================

# STEP 1: Parse logs using Regex into structured columns

# ============================================================

log_pattern = r'(\S+) \S+ \S+ \[([^\]]+)\] "(\S+) (\S+) \S+" (\d+) (\d+)'

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

parsed = (spark.sparkContext.parallelize(log_data)

    .map(lambda line: re.match(log_pattern, line))

    .filter(lambda m: m is not None)

    .map(lambda m: (m.group(1), m.group(2), m.group(3),

                    m.group(4), int(m.group(5)), int(m.group(6)))))

schema = StructType([

    StructField("ip",          StringType()),

    StructField("timestamp",   StringType()),

    StructField("method",      StringType()),

    StructField("endpoint",    StringType()),

    StructField("status_code", IntegerType()),

    StructField("bytes",       IntegerType()),

])

df = spark.createDataFrame(parsed, schema)

print("=" * 55)

print("   LOG ANALYSIS DASHBOARD - Python + PySpark")

print("=" * 55)

# ============================================================

# STEP 2: Total requests

# ============================================================

total = df.count()

print(f"\n[1] Total Requests: {total}")

# ============================================================

# STEP 3: Status code distribution

# ============================================================

print("\n[2] Status Code Distribution:")

df.groupBy("status_code").count().orderBy("status_code").show()

# ============================================================

# STEP 4: Error rate (status >= 400)

# ============================================================

errors = df.filter(col("status_code") >= 400).count()

error_rate = round((errors / total) * 100, 2)

print(f"[3] Error Rate: {error_rate}%  ({errors} errors / {total} total)")

# ============================================================

# STEP 5: Top IPs by request volume

# ============================================================

print("\n[4] Top IPs by Request Volume:")

df.groupBy("ip").count().orderBy(desc("count")).limit(3).show()

# ============================================================

# STEP 6: Most visited endpoints

# ============================================================

print("\n[5] Most Visited Endpoints:")

df.groupBy("endpoint").count().orderBy(desc("count")).limit(5).show()

# ============================================================

# STEP 7: HTTP method breakdown

# ============================================================

print("\n[6] HTTP Method Breakdown:")

df.groupBy("method").count().orderBy(desc("count")).show()

# ============================================================

# STEP 8: Total data transferred

# ============================================================

total_bytes = df.agg(spark_sum("bytes").alias("total")).collect()[0]["total"]

print(f"[7] Total Data Transferred: {total_bytes:,} bytes  ({round(total_bytes/1024, 2)} KB)")

spark.stop()

print("\n Analysis Complete!")

```

Learning Outcomes

After completing this project, you will be able to:

Parse unstructured log data using regex patterns

Transform RDDs into DataFrames with explicit schemas

Perform complex aggregations and groupby operations

Extract meaningful insights from log data

Apply data processing techniques that scale to production workloads

Requirements

Python 3.6 or higher

PySpark 2.4 or later

Basic understanding of Apache logs format

Familiarity with Spark DataFrames and SQL operations

Generated from: PySpark.in Log Analysis Dashboard Project

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges