PySpark: Beginner to Advanced

A structured path through the PySpark challenge catalog: DataFrame basics, core transformations, aggregation and joins, window functions, data cleaning, then advanced and real-world pipeline scenarios.

Sign in to start this path and track your progress.

Stages

1. 1DataFrame Basics Not Startedselect, filter and withColumn on a DataFrame.▲
   →Select Two Columns from a DataFrameEasy○Filter Rows with a ConditionEasy○Add a Bonus Column with withColumnEasy○Sort by Salary DescendingEasy
2. 2DataFrame Operations Not StartedSorting, top-N filtering, and fixing a broken transformation pipeline.▼
3. 3Aggregations & GroupBy Not StartedgroupBy/agg patterns for per-customer and per-region summaries.▼
4. 4Joins Not StartedJoining DataFrames. The catalog currently has one published PySpark Joins challenge outside the other stages -- reported as a content gap, not forced.▼
5. 5Window Functions Not StartedRunning totals, day-over-day change, and moving averages with Window specs.▼
6. 6Data Cleaning & Deduplication Not StartedRemoving duplicate records and parsing/cleaning raw text fields.▼
7. 7Advanced & Real-World Pipeline Scenarios Not StartedSCD Type 2, incremental/CDC loads, broadcast joins and pivoting -- applied, production-flavored problems. Several optional items here are premium.▼