PySpark groupBy(): group and aggregate a DataFrame

PySpark `groupBy()` groups the rows of a DataFrame by one or more columns so you can compute aggregates (count, sum, avg, min, max) per group. It returns a `GroupedData` object; you then call `.agg()` or a shortcut like `.count()` / `.sum()` to produce the aggregated DataFrame.

Basic syntax

Call `df.groupBy("col")` and chain an aggregation. Use `.agg()` with functions from `pyspark.sql.functions` to compute several aggregates at once and alias the output columns.

Group by multiple columns

Pass several column names to `groupBy("col1", "col2")` to aggregate within each unique combination. This is the DataFrame equivalent of SQL `GROUP BY col1, col2`.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, avg, count

spark = SparkSession.builder.getOrCreate()
orders = spark.createDataFrame(
    [(101, 250.0), (102, 300.0), (101, 150.0), (103, 400.0), (102, 200.0)],
    ["customer_id", "order_value"],
)

result = (
    orders.groupBy("customer_id")
          .agg(
              sum("order_value").alias("total_sales"),
              avg("order_value").alias("avg_order_value"),
              count("*").alias("num_orders"),
          )
)
result.show()

Groups orders by customer_id, then computes total, average and count of order_value per customer.

Run this example in the free online PySpark compiler

Frequently asked questions

What does PySpark groupBy() return?

It returns a GroupedData object, not a DataFrame. You must chain an aggregation such as .agg(), .count(), .sum() or .avg() to get back a DataFrame.

How do I group by multiple columns in PySpark?

Pass all the columns to groupBy, e.g. df.groupBy("region", "product").agg(sum("amount")). It aggregates within each unique combination of the columns.

What is the difference between groupBy and agg in PySpark?

groupBy defines the grouping keys and returns GroupedData; agg applies one or more aggregate functions to each group and returns the aggregated DataFrame.

Can I run PySpark groupBy online without installing Spark?

Yes — you can run this example free in the browser on the PySpark.in online compiler; it uses a real Spark session with no setup.

Practice challenges

• Calculate Total Sales and Average Order Value by Customer
• Analyze Sales Data to Find Top Performers by Region

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs