Pandas
Python tutorial · PySpark.in
PANDAS
Pandas is a Python library used for data cleaning, manipulation, analysis, and exploration.It provides fast, flexible, and powerful DataFrames similar to Excel tables.
- An open-source library for handling structured data.
- Built on top of NumPy.
Uses two main data structures:
- Series for 1D labeled array
- DataFrame for 2D labeled table (rows + columns)
Series for 1D labeled array.
How to Import?
```
import pandas as pd
```
Pandas Data Structures
Pandas Series
A Series in pandas is a one-dimensional data structure similar to a column in Excel or a list with labels.
The values are 10, 20, 30
The indexes (labels) are "a", "b", "c"
```
import pandas as pd
s = pd.Series([10, 20, 30], index=["a","b","c"])
print(s)
```
Pandas DataFrame
A DataFrame is a two-dimensional table-like structure (rows & columns), similar to an Excel sheet or SQL table.
```
import pandas as pd
df = pd.DataFrame({
"Name": ["A", "B"],
"Marks": [90, 80]
})
print(df)
```
- It creates 2 columns: Name and Marks
- Each row represents one record
Read and Write Data (I/O)
Read Files
```
hide
import pandas as pd
df = pd.read_csv("data.csv")
df = pd.read_excel("file.xlsx")
df = pd.read_json("file.json")
```
These commands are used to load data from different file formats into a DataFrame.
Once loaded, df contains the complete dataset in table form.
Write Files
```
Hide
import pandas as pd
df.to_csv("out.csv", index=False)
df.to_excel("out.xlsx", index=False)
```
- These are Used to export and save a DataFrame to files.
- index=False ensures that row numbers are not saved as a separate column.
Basic DataFrame Operations
Check first/last rows
```
import pandas as pd
df.head()
df.tail()
print(df.head())
print(df.tail())
```
- head() shows the first 5 rows
- tail() shows the last 5 rows
- Useful for quickly previewing a dataset.
Shape (rows, columns)
```
import pandas as pd
df.shape
print(df.shape)
```
Returns (rows, columns)which will helps to know dataset size.
Example: (2, 2) has 2 rows and 2 columns.
Columns
```
import pandas as pd
df.columns
print(df.columns)
```
Shows the list of column headers present in the DataFrame.
Info
```
import pandas as pd
df.info()
print(df.info)
```
Displays important details:
- Number of rows and columns
- Column names & data types
- Count of non-null values (missing or not)
- Memory usage
Very useful for dataset overview and data cleaning.
Summary statistics
```
import pandas as pd
df.describe()
print(df.describe)
```
describe() returns statistical summary of numerical columns such as count, mean, min, max, and standard deviation. Helps to quickly understand dataset distribution and numeric behavior.
Selecting Data
Select a single column
```
import pandas as pd
# Create DataFrame
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [25, 30, 22]
})
# Accessing column
print(df["Age"]) # Method 1
print(df.Age) # Method 2
```
Select multiple columns
```
import pandas as pd
# Create DataFrame
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [25, 30, 22],
"City": ["Delhi", "Mumbai", "Kolkata"]
})
# Select multiple columns
print(df[["Name", "Age"]])
```
Provides a subset DataFrame containing only selected columns used when you need specific fields only.
Select rows by index
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [25, 30, 22],
"City": ["Delhi", "Mumbai", "Kolkata"]
})
print(df.loc[0]) # label-based, row with index 0
print(df.iloc[0]) # position-based, first row
```
- loc selects based on index label
- iloc selects based on numeric position
Both return the entire first row here.
Select rows and columns together
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [25, 30, 22],
"City": ["Delhi", "Mumbai", "Kolkata"]
})
print(df.loc[0, "Name"]) # row 0, column Name
print(df.iloc[0, 2]) # row 0, column index 2 (City)
```
Helps to extract a specific cell value from the DataFrame.
Filtering / Conditional Selection
Filter rows with a condition
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rahul"],
"Age": [25, 30, 22, 19],
"Gender": ["Male", "Female", "Female", "Male"],
"City": ["Delhi", "Mumbai", "Kolkata", "Delhi"]
})
print(df[df["Age"] > 20])
print(df[(df["Age"] > 18) & (df["Gender"] == "Male")])
print(df[df["City"].isin(["Delhi", "Mumbai"])])
print(df[df["Name"].str.contains("A", case=False)])
```
- Shows only rows where Age is greater than 20.
- Returns rows where Age > 18 AND Gender = Male.
Use & for AND, | for OR. - Selects rows where City is either Delhi or Mumbai.
- Returns rows where Name contains letter “A” (case-insensitive).
Adding / Updating Columns
Add new column
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Math": [80, 90, 75],
"Sci": [70, 85, 95]
})
df["Total"] = df["Math"] + df["Sci"]
print(df)
```
Update values
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [20, 22, 25],
"City": ["Delhi", "Mumbai", "Pune"]
})
print(df)
df["Age"] = df["Age"] + 1
print(df)
```
Deleting Data
Delete column
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [20, 22, 25],
"City": ["Delhi", "Mumbai", "Pune"]
})
print(df)
df.drop("Age", axis=1, inplace=True)
print(df)
```
Delete rows
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha"],
"Age": [20, 22, 25],
"City": ["Delhi", "Mumbai", "Pune"]
})
print(df)
df.drop(0, axis=0, inplace=True) # deletes first row
print(df)
```
- Creates a new column using operations on existing columns.
- Modifies values of an existing column are useful for updating or transforming data.
- Removes the Age column permanently.
- Removes the row where index = 0.
Handling Missing Values
Check missing values
```
import pandas as pd
import numpy as np
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", None],
"Age": [20, 22, None, 25],
"City": ["Delhi", None, "Pune", "Mumbai"]
})
print(df)
df.isnull().sum()
```
Counts how many NaN / None values each column contains.
Drop Rows with Null Values
```
import pandas as pd
import numpy as np
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", None],
"Age": [20, 22, None, 25],
"City": ["Delhi", None, "Pune", "Mumbai"]
})
print(df)
df.dropna()
```
Removes all rows containing at least one null value.
Fill nulls
```
import pandas as pd
import numpy as np
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", None],
"Age": [20, 22, None, 25],
"City": ["Delhi", None, "Pune", "Mumbai"]
})
print(df)
df.fillna(0)
df["Age"].fillna(df["Age"].mean(), inplace=True)# Fill Age with mean value
df["City"].fillna("Unknown", inplace=True)# Fill City with a specific value
df["Name"].fillna("NoName", inplace=True)# Fill Name with "NoName"
```
Replaces missing values instead of deleting rows — useful for preserving data.
Sorting
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df.sort_values("Age")# Sort by one column (Age)
df.sort_values(["Age", "Marks"], ascending=[True, False])# Sort by two columns (Age ascending, Marks descending)
```
- Sorts the rows based on Age in ascending order (default).
- Sorts by Age (ascending) and if Age is same, sorts by Marks (descending).
Grouping and Aggregation
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df.groupby("City")["Marks"].mean()#Group by City → Average Marks
```
Groups data by City which shows average Marks for each city.
Group by Gender: Multiple Aggregations
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df.groupby("Gender").agg({
"Marks": ["mean", "max", "min"],
"Age": "count"
})
```
Gives mean, max, min of Marks and count of Age for each Gender.
Merging, Joining & Concatenation
Concatenate (stack)
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print("Original DF:\n", df)
# Split into two DataFrames
df1 = df.iloc[:3] # First 3 rows
df2 = df.iloc[3:] # Remaining rows
print("\nDF1:\n", df1)
print("\nDF2:\n", df2)
# Concatenate (stack row-wise)
result_concat = pd.concat([df1, df2])
print("\nConcatenated Result:\n", result_concat)
```
Combines two dataframes vertically (row-wise).
Note: df1 must be created before use can fix indentation
Merge (SQL-style join)
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df_city = pd.DataFrame({
"City": ["Delhi", "Mumbai", "Pune"],
"State": ["Delhi", "Maharashtra", "Maharashtra"]
})
result_merge = pd.merge(df, df_city, on="City")
print(result_merge)# Let’s add a new DataFrame to merge (same City key)
```
Merges two DataFrames based on City column. Works like an INNER JOIN by default.
Join Type | Meaning |
|---|---|
inner | Only matching rows |
left | All rows from left + matched from right |
right | All rows from right + matched from left |
outer | All rows from both (unmatched = NaN) |
Left, right, inner, outer join
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df_sch = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Sima"],
"Scholarship": ["Yes", "No", "Yes", "Yes"]
})
# left join (all rows from df, scholarship only if matched)
result_left = pd.merge(df, df_sch, on="Name", how="left")
print(result_left)# (Joining student DataFrame with a dataframe of scholarship)
```
Example left join result: Shows all students, scholarship only if available.
Pivot Tables
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
pivot = df.pivot_table(values="Marks", index="City", columns="Gender", aggfunc="mean")
print(pivot)# This gives the average Marks for each gender in each city.
```
Creates a summary table showing average marks for each gender in each city.
Structure = rows (City) × columns (Gender)
Apply Functions
Using apply():
Apply function on a column or row.
- Used for Series or row-wise operations.
- Adds 2 bonus marks to every value in the Marks column.
- Best when you want to apply a function on one column (or row using axis=1).
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df["New"] = df["Marks"].apply(lambda x: x + 2)
print(df)# Add 2 marks bonus to each "Marks" entry
```
Using applymap() :
Apply function on every cell
- Multiplies every numeric value of the DataFrame by 2.
Non-numeric cells remain unchanged. - Use carefully it affects all cells.
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df2 = df.applymap(lambda x: x*2 if isinstance(x, (int, float)) else x)
print(df2)# Multiply every value by 2
df["Total"] = df.apply(lambda row: row["Age"] + row["Marks"], axis=1)# Calculate a Total Score = Age + Marks
```
Note: Non-numeric cells remain unchanged
applymap() acts on every cell, so use carefully.
Using map() :
Works only for Series
```
import pandas as pd
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
df["Gender"] = df["Gender"].map({"M": "Male", "F": "Female"})
print(df)# Convert "M" / "F" to "Male" / "Female"
```
- Converts codes to labels (M → Male, F → Female).
- Perfect for value mapping or replacements.
Handling Dates in Pandas
```
import pandas as pd
# Creating the DataFrame
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"],
"Date": ["2024-01-12", "2023-09-18", "2022-05-10", "2024-03-02", "2023-12-20"]
})
# Converting Date column to datetime format
df["Date"] = pd.to_datetime(df["Date"])
# Extracting Year, Month, Day
df["Year"] = df["Date"].dt.year
df["Month"] = df["Date"].dt.month_name()
df["Day"] = df["Date"].dt.day
print(df)
```
Now convert to datetime & extract pieces
```
import pandas as pd
# Creating DataFrame
df = pd.DataFrame({
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["M", "F", "F", "M", "F"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"],
"Date": ["2024-01-12", "2023-09-18", "2022-05-10", "2024-03-02", "2023-12-20"]
})
# Convert Gender short form to full name
df["Gender"] = df["Gender"].map({"M": "Male", "F": "Female"})
# Convert to datetime
df["Date"] = pd.to_datetime(df["Date"])
# Extract date parts
df["Year"] = df["Date"].dt.year
df["Month"] = df["Date"].dt.month_name() # Full month name
df["Day"] = df["Date"].dt.day
print(df)
```
Indexing Operations
Set index:
Make a column the index
```
import pandas as pd
df = pd.DataFrame({
"ID": [101, 102, 103, 104, 105],
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["Male", "Female", "Female", "Male", "Female"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df.set_index("ID", inplace=True)# Set "ID" column as index
print(df)
```
- "ID" becomes the new index
It removes "ID" from normal columns (since it is now the index)
Reset index:
Convert index back to a column. Index becomes a normal column
New index becomes 0,1,2,3…
```
import pandas as pd
df = pd.DataFrame({
"ID": [101, 102, 103, 104, 105],
"Name": ["Aman", "Rekha", "Megha", "Rohan", "Sima"],
"Age": [20, 22, 21, 23, 22],
"Marks": [85, 90, 78, 88, 92],
"Gender": ["Male", "Female", "Female", "Male", "Female"],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Mumbai"]
})
print(df)
df.reset_index(inplace=True)# Convert index back into a normal column
print(df)
```
More Python tutorials
- What is Python and Why is it used for Data Science and Data Engineering?
- How Does Python Work in the Backend? Internal Working of Python
- Top 30 Python Interview Questions for Data Science
- 3-Month Python Roadmap to Excel in Data Science and Machine Learning
- Python Data Types Explained – A Beginner’s Guide
- test
All tutorials · Try the free PySpark compiler · Practice challenges