Apache Spark 3.x
Process billions of rows in minutes — the distributed computing engine every data engineer must know.
Start Learning Apache Spark 3.xWhat You'll Learn
- What Spark is, why it's fast, and how distributed computing works
- How to create a SparkSession and load data from files
- Core DataFrame operations: select, filter, groupBy, join, withColumn
- Spark SQL — writing SQL queries directly on DataFrames
- How Spark's lazy evaluation and DAG execution model works
- Common transformations and actions in PySpark
- Performance tuning: partitioning, caching, broadcast joins
- Reading from and writing to Parquet, CSV, JSON, and Delta Lake
Introduction to Apache Spark 3.x
Apache Spark is an open-source distributed computing framework used to process huge amounts of data — think terabytes or petabytes — across a cluster of computers. Instead of running your code on one machine, Spark splits the data and the computation across many machines working in parallel, making it dramatically faster than traditional tools for large datasets.
Spark 3.x introduced major improvements including Adaptive Query Execution (AQE) — which automatically optimises query plans at runtime — and dynamic partition pruning, making queries on large partitioned datasets 10-100x faster. It also added built-in support for Pandas API on Spark (formerly Koalas), so data scientists familiar with pandas can scale their code to big data without rewriting everything.
You interact with Spark through the DataFrame API (similar to pandas but distributed), Spark SQL (write standard SQL against big data), or the lower-level RDD API. PySpark is the Python interface to Spark and is the most popular way to use it in modern data engineering teams alongside tools like Databricks, AWS EMR, and Azure Synapse.
Video Tutorials
Handpicked free YouTube videos to accelerate your understanding
PySpark Tutorial for Beginners — Full Course
Start from zero — install PySpark, create your first SparkSession, and master DataFrames, transformations, and Spark SQL.
Apache Spark Architecture Explained
Understand how Spark distributes work across a cluster — drivers, executors, DAGs, stages, and shuffle operations explained visually.
Getting started with PySpark DataFrames
Copy the code below and paste it into your Python environment or our free online compiler.
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, avg, count, when
# 1. Create a SparkSession (entry point to Spark)
spark = SparkSession.builder \
.appName("MyFirstSparkJob") \
.getOrCreate()
# 2. Create a sample DataFrame (in real life, read from S3/HDFS)
data = [
("Alice", "Engineering", 95000),
("Bob", "Engineering", 88000),
("Carol", "Marketing", 72000),
("David", "Marketing", 68000),
("Eve", "Engineering", 102000),
]
df = spark.createDataFrame(data, ["name", "department", "salary"])
# 3. Show the data
df.show()
# 4. Filter rows where salary > 85000
high_earners = df.filter(col("salary") > 85000)
high_earners.show()
# 5. Group by department and calculate average salary
dept_avg = df.groupBy("department") \
.agg(avg("salary").alias("avg_salary"),
count("*").alias("headcount"))
dept_avg.show()
# +------------+-----------+---------+
# | department|avg_salary |headcount|
# +------------+-----------+---------+
# | Engineering| 95000.0 | 3|
# | Marketing| 70000.0 | 2|
# +------------+-----------+---------+
# 6. Write results to Parquet
dept_avg.write.mode("overwrite").parquet("/output/dept_stats")Key Concepts Explained
Master these terms and you'll understand 80% of the conversations in this field.
SparkSession
The main entry point for all Spark functionality. You create one at the start of every Spark application. It lets you read data, run SQL, and configure the cluster.
DataFrame
A distributed table of data organised into named columns. Similar to a pandas DataFrame or a SQL table, but the data is split across many machines.
Transformation vs Action
Transformations (filter, select, groupBy) are lazy — they build a plan but don't run. Actions (show, count, write) trigger execution. This allows Spark to optimise the full query plan.
DAG (Directed Acyclic Graph)
Spark builds a graph of all your transformations before executing. This lets it optimise operations, skip unnecessary work, and retry failed tasks automatically.
Partition
Spark splits data into chunks called partitions. Each partition is processed by one executor. More partitions = more parallelism. Repartition/coalesce controls the number.
Catalyst Optimizer
Spark's query optimizer that rewrites your logical query plan into the most efficient physical plan. AQE in Spark 3 makes this happen at runtime with real stats.
Shuffle
When Spark needs to redistribute data across nodes (e.g., for a join or groupBy), it performs a shuffle. Shuffles are expensive — minimising them is key to Spark performance.
Parquet
A columnar file format optimised for big data. It compresses well, supports predicate pushdown, and is 5-10x faster to query than CSV for analytical workloads.
Your Apache Spark 3.x Learning Path
Follow these steps in order — each one builds on the last. Designed for complete beginners.
- 1
Python Basics
Learn Python lists, dictionaries, functions, and list comprehensions. All Spark code in PySpark is written in Python.
- 2
Install PySpark Locally
Install Java 11 and PySpark via pip. Run your first SparkSession locally on your laptop to process sample data.
- 3
DataFrame Operations
Master select, filter, withColumn, groupBy, join, and sort. These 6 operations cover 90% of real data engineering work.
- 4
Spark SQL
Register DataFrames as temporary views and write SQL queries. Spark SQL is often easier to read for complex transformations.
- 5
File Formats & Data Sources
Read and write Parquet, JSON, CSV, and Delta Lake. Understand partitioning by date or category to speed up reads.
- 6
Performance Tuning
Learn about broadcast joins, caching, partition skew, and adaptive query execution to make your jobs 10x faster.
- 7
Spark on the Cloud
Deploy jobs on Databricks, AWS EMR, or Azure Synapse. Use job clusters and S3/ADLS for production workflows.
Ready to master Apache Spark 3.x?
Explore our free tutorials, hands-on code examples, and interview questions. No sign-up. No paywalls. Forever free.