PySpark Window Functions (Beginner Friendly Guide)
Spark tutorial · PySpark.in
## 1) What are Window Functions in PySpark?
Window functions let you calculate values across related rows **without** collapsing rows like `groupBy`. This is great for ranking, running totals, moving averages, and comparing the current row with previous/next rows.
---
## 2) Sample Data
We will use a simple sales dataset.
```python
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.window import Window
spark = SparkSession.builder.getOrCreate()
data = [
("u1", "o1", "2024-01-01", 100),
("u1", "o2", "2024-01-05", 200),
("u1", "o3", "2024-01-10", 150),
("u2", "o4", "2024-01-03", 80),
("u2", "o5", "2024-01-08", 120),
]
df = spark.createDataFrame(
data, ["user_id", "order_id", "order_date", "amount"]
).withColumn("order_date", F.to_date("order_date"))
```
---
## 3) Define a Window
A window = **partition + order**.
```python
w = Window.partitionBy("user_id").orderBy("order_date")
```
---
## 4) Row Number / Rank / Dense Rank
```python
df_rank = df \
.withColumn("row_num", F.row_number().over(w)) \
.withColumn("rank", F.rank().over(w)) \
.withColumn("dense_rank", F.dense_rank().over(w))
```
**Use case:** Find 1st, 2nd, 3rd order per user.
---
## 5) Running Total
```python
w_rt = Window.partitionBy("user_id").orderBy("order_date") \
.rowsBetween(Window.unboundedPreceding, Window.currentRow)
df_rt = df.withColumn("running_total", F.sum("amount").over(w_rt))
```
**Use case:** Total spend so far for each user.
---
## 6) Moving Average (Last 3 Orders)
```python
w_ma = Window.partitionBy("user_id").orderBy("order_date") \
.rowsBetween(-2, 0)
df_ma = df.withColumn("moving_avg_3", F.avg("amount").over(w_ma))
```
**Use case:** Smooth out spending trend.
---
## 7) Lag & Lead (Previous / Next Row)
```python
df_lag = df \
.withColumn("prev_amount", F.lag("amount", 1).over(w)) \
.withColumn("next_amount", F.lead("amount", 1).over(w))
```
**Use case:** Compare current order with previous order.
---
## 8) Top N per Group (Example)
Get the **latest 1 order** per user.
```python
latest = df \
.withColumn("row_num", F.row_number().over(w.orderBy(F.col("order_date").desc()))) \
.filter(F.col("row_num") == 1)
```
---
## 9) Tips (Very Important)
- Always add **ORDER BY** if the order matters.
- Use **rowsBetween** for rolling windows.
- Window functions can be expensive → partition wisely.
---
## 10) Practice (Try Yourself)
1) Top 2 orders per user
2) Running total by date
3) Difference between current and previous order
4) Find max order per user
---
### Conclusion
Window functions are one of the most powerful tools in PySpark. Once you master them, you can build clean analytics, faster ETL pipelines, and deeper insights.
More Spark tutorials
- about pyspark
- text diagram
- Apache Spark Runtime Architecture
- Introduction to RDD
- Actions vs Transformations
- Lazy Evaluation in PySpark
All tutorials · Try the free PySpark compiler · Practice challenges