Grouping & Aggregation
Python tutorial · PySpark.in
What is Grouping & Aggregation?
Grouping and Aggregation is the process of splitting data into groups based on one or more columns and then applying summary functions to each group.
Example:
- City-wise average marks
- Department-wise total salary
- Gender-wise count
Why do we need GroupBy?
✔ Data summarization
✔ Pattern discovery
✔ Business insights
✔ Feature engineering
✔ Interview favorite topic
Sample DataFrame (Common for all examples)
```
import pandas as pd
data = {
"Name": ["Megha", "Amit", "Riya", "Neha", "Rahul", "Ankit"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Delhi", "Mumbai"],
"Department": ["IT", "HR", "IT", "IT", "HR", "HR"],
"Marks": [85, 90, 88, 92, 80, 75]
}
df = pd.DataFrame(data)
print(df)
```
- groupby(): splits the data into groups based on unique values of a column.
Practical: Group by City
```
grouped = df.groupby("City")
print(grouped)
```
Why This Output Appears?
groupby() does not immediately perform any calculation.
It only creates a GroupBy object that stores:
- grouping information
- column references
- indices of each group
That’s why Python prints:
DataFrameGroupBy object
aggregation
```
print(df.groupby("City")["Marks"].mean())
```
Aggregation Functions
(sum, mean, count, min, max)
Aggregation functions summarize data into a single value.
Practical Examples
```
print(df.groupby("Department")["Marks"].sum())
```
```
print(df.groupby("Department")["Marks"].count())
```
```
print(df.groupby("Department")["Marks"].max())
```
```
print(df.groupby("Department")["Marks"].min())
```
Multiple Aggregations
Apply multiple aggregation functions on grouped data. Very important for EDA & reports
Practical
```
print(df.groupby("City")["Marks"].agg(["mean", "max", "min", "count"]))
```
agg() Function
agg() allows custom and multiple aggregations in one step.
Practical
```
print(
df.groupby("Department").agg(
Avg_Marks=("Marks", "mean"),
Total=("Marks", "sum"),
Highest=("Marks", "max")
)
)
```
Group Filtering
Group filtering keeps only those groups that satisfy a condition. Used in advanced analytics
Practical
Keep departments with average marks > 85
```
filtered = df.groupby("Department").filter(
lambda x: x["Marks"].mean() > 85
)
print(filtered)
```
Group Transformation
Group transformation returns same size output as original data, unlike aggregation. Very powerful for feature engineering
Practical
Add column: difference from group mean
```
df["Marks_Diff"] = df.groupby("City")["Marks"].transform(
lambda x: x - x.mean()
)
print(df)
```
GroupBy vs Transform vs Filter
Method | Output Size | Use |
|---|---|---|
agg | Reduced | Summary |
transform | Same | New column |
filter | Reduced | Group selection |
More Python tutorials
- What is Python and Why is it used for Data Science and Data Engineering?
- How Does Python Work in the Backend? Internal Working of Python
- Top 30 Python Interview Questions for Data Science
- 3-Month Python Roadmap to Excel in Data Science and Machine Learning
- Python Data Types Explained – A Beginner’s Guide
- test
All tutorials · Try the free PySpark compiler · Practice challenges