PySpark groupBy(): group and aggregate a DataFrame

PySpark `groupBy()` groups the rows of a DataFrame by one or more columns so you can compute aggregates (count, sum, avg, min, max) per group. It returns a `GroupedData` object; you then call `.agg()` or a shortcut like `.count()` / `.sum()` to produce the aggregated DataFrame.

Basic syntax

Call `df.groupBy("col")` and chain an aggregation. Use `.agg()` with functions from `pyspark.sql.functions` to compute several aggregates at once and alias the output columns.

Group by multiple columns

Pass several column names to `groupBy("col1", "col2")` to aggregate within each unique combination. This is the DataFrame equivalent of SQL `GROUP BY col1, col2`.

groupBy with agg, and naming your outputs

groupBy returns a grouped object that does nothing until you aggregate it. The flexible form is agg(), which takes any number of aggregate expressions: df.groupBy('dept').agg(F.count('*').alias('headcount'), F.avg('salary').alias('avg_salary'), F.max('salary').alias('top_salary')). Always alias the results, otherwise you get column names like 'avg(salary)' that are awkward to reference. You can group by several columns by passing them all, and the shortcut forms such as groupBy('dept').count() exist for the simple cases.

Nulls, counting correctly, and pivoting

count('*') counts rows while count('col') counts non-null values in that column, so the two disagree exactly where your data is missing — a useful accident for data quality checks, and a silent bug if you did not intend it. countDistinct() is exact but expensive on high-cardinality columns; approx_count_distinct() trades a small error for a large speed-up. Unlike a join, grouping treats nulls as a single group rather than discarding them. To turn distinct values into columns, add pivot: df.groupBy('product').pivot('month').sum('sales'). Supplying the list of pivot values explicitly avoids an extra pass over the data to discover them.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, avg, count

spark = SparkSession.builder.getOrCreate()
orders = spark.createDataFrame(
    [(101, 250.0), (102, 300.0), (101, 150.0), (103, 400.0), (102, 200.0)],
    ["customer_id", "order_value"],
)

result = (
    orders.groupBy("customer_id")
          .agg(
              sum("order_value").alias("total_sales"),
              avg("order_value").alias("avg_order_value"),
              count("*").alias("num_orders"),
          )
)
result.show()

Groups orders by customer_id, then computes total, average and count of order_value per customer.

Run this example in the free online PySpark compiler

Frequently asked questions

What does PySpark groupBy() return?

It returns a GroupedData object, not a DataFrame. You must chain an aggregation such as .agg(), .count(), .sum() or .avg() to get back a DataFrame.

How do I group by multiple columns in PySpark?

Pass all the columns to groupBy, e.g. df.groupBy("region", "product").agg(sum("amount")). It aggregates within each unique combination of the columns.

What is the difference between groupBy and agg in PySpark?

groupBy defines the grouping keys and returns GroupedData; agg applies one or more aggregate functions to each group and returns the aggregated DataFrame.

Can I run PySpark groupBy online without installing Spark?

Yes — you can run this example free in the browser on the PySpark.in online compiler; it uses a real Spark session with no setup.

Practice challenges

• Calculate Total Sales and Average Order Value by Customer
• Analyze Sales Data to Find Top Performers by Region

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs