psypark Cheat Sheet

Published 2026-07-09 in Data Engineering

Most people know what Pivot does. But when asked to write it in PySpark without Googling? Blank. Here is the full breakdown ↓ → What is Pivot? Pivot transforms row values into column headers. Converts long format data into wide format. → What is Unpivot? Unpivot does the opposite. Converts column headers back into row values. Wide format → long format. → When to Use Each Use Pivot when: • You want to compare values ↳ across categories side by side • Creating summary reports ↳ with months or categories as columns • Feeding data into dashboards ↳ that expect wide format Use Unpivot when: • Source data has dynamic columns ↳ that need to become rows • Normalizing data before loading ↳ into a data warehouse • Feeding into ML pipelines ↳ that expect long format → Pivot in PySpark from pyspark.sql import functions as F df.groupBy("product") \ .pivot("month") \ .agg(F.sum("sales")) Input: ↳ product | month | sales ↳ A | Jan | 100 ↳ A | Feb | 200 ↳ B | Jan | 150 Output: ↳ product | Jan | Feb ↳ A | 100 | 200 ↳ B | 150 | null → Pivot with Specific Values (Faster) df.groupBy("product") \ .pivot("month", ["Jan", "Feb", "Mar"]) \ .agg(F.sum("sales")) ↳ Always specify pivot values explicitly ↳…

More Data Engineering articles · All collections · Practice challenges