Learn PySpark, SQL & Data Engineering

PySpark is the Python API for Apache Spark, the engine most data teams reach for once their data outgrows a single machine. Learning it means learning three things at once: the DataFrame API you write, the SQL that expresses the same ideas declaratively, and the way Spark distributes the work underneath. The guides below take them one at a time, each with code you can run in the browser compiler without installing anything. Nothing here assumes you have a cluster: every example runs on a single machine, which is enough to learn what the operations mean before you meet data large enough to punish getting them wrong.

Start with the DataFrame API

Most PySpark work is transforming DataFrames. Begin with select and filter to shape rows and columns, withColumn to derive new ones, then groupBy and join to combine data. Reading data comes up immediately too, which is what read.csv covers, along with fillna and dropDuplicates for the cleaning that always follows.

Then the parts that separate beginners from practitioners

Window functions answer questions about a row in the context of its neighbours — running totals, rankings, gaps between events — and they have a direct SQL equivalent worth knowing. repartition versus coalesce is where people first meet Spark's execution model, and UDFs are where they first pay for ignoring it. On the SQL side, common table expressions keep long queries readable, and orderBy is deceptively expensive at scale.

Where PySpark fits in a data platform

Spark rarely runs alone. Spark SQL is the same engine reached through queries; Structured Streaming applies it to unbounded data with watermarks and event-time windows; Delta Lake adds ACID transactions and time travel on top of the files Spark writes; and MLlib trains models on the same distributed data. If you are earlier than that, start with what data engineering actually is and ETL versus ELT.

Practise what you read

Reading about a shuffle is not the same as causing one. Each guide pairs with the coding challenges, which are graded against real Spark output, and with interview questions if you are preparing for a data engineering role.

All guides

• PySpark groupBy(): group and aggregate a DataFrame — PySpark `groupBy()` groups the rows of a DataFrame by one or more columns so you can compute aggregates (count, sum, avg
• PySpark join(): combine two DataFrames — PySpark `join()` combines two DataFrames on one or more key columns. You control the join type with the `how` argument —
• PySpark window functions explained — PySpark window functions compute a value across a set of rows related to the current row without collapsing them (unlike
• SQL window functions with examples — SQL window functions perform a calculation across a set of rows related to the current row using the `OVER()` clause, wi
• ETL vs ELT: what's the difference? — ETL (Extract, Transform, Load) transforms data **before** loading it into the destination; ELT (Extract, Load, Transform
• PySpark withColumn(): add or transform a column — PySpark `withColumn(name, expr)` returns a new DataFrame with a column added or replaced. The second argument is a Colum
• PySpark filter() and where(): keep rows that match a condition — PySpark `filter()` (and its alias `where()`) returns a new DataFrame keeping only the rows that satisfy a Boolean condit
• PySpark select(): pick and compute columns — PySpark `select()` returns a new DataFrame containing only the columns (or column expressions) you name. You can pass co
• PySpark: read a CSV into a DataFrame — In PySpark you load a CSV with `spark.read.csv(path)` or `spark.read.option(...).csv(path)`. Set `header=True` to use th
• PySpark orderBy() and sort(): order rows — PySpark `orderBy()` (alias `sort()`) returns a new DataFrame with rows sorted by one or more columns. By default sorting
• PySpark dropDuplicates() and distinct(): deduplicate rows — PySpark `distinct()` removes fully duplicate rows; `dropDuplicates(subset)` removes rows that are duplicates on a chosen
• PySpark UDFs: custom functions on columns — A PySpark UDF wraps a Python function so it can run on DataFrame columns. Define it with `udf(fn, returnType)` or the `@
• SQL CTEs: the WITH clause — A SQL CTE (Common Table Expression) is a named temporary result set defined with the `WITH` clause and referenced later
• repartition() vs coalesce() in PySpark — Both `repartition()` and `coalesce()` change the number of partitions of a PySpark DataFrame. `repartition(n)` does a fu
• PySpark fillna(): replace null values — PySpark `fillna(value)` (alias `df.na.fill(value)`) replaces null values in a DataFrame. Pass a single value to fill all
• What is Data Engineering? — Data Engineering is the practice of designing, building and maintaining the systems that collect, store, transform and s