Data Manupulation
Python tutorial · PySpark.in
What is Data Manipulation?
Data Manipulation is the process of modifying, transforming, and restructuring data to make it suitable for analysis and machine learning. This step comes after cleaning & preprocessing
Sample DataFrame Used
```
import pandas as pd
data = {
"Name": ["Megha", "Amit", "Riya", "Neha", "Rahul"],
"Age": [21, 22, 23, 21, 22],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Delhi"],
"Marks": [85, 90, 88, 92, 80]
}
df = pd.DataFrame(data)
print(df)
```
Adding Columns
Adding columns means creating new features from existing data. It is very important for feature engineering.
Practical 1: Add Constant Column
```
df["Country"] = "India"
print(df)
```
Practical 2: Derived Column
```
df["Result"] = df["Marks"].apply(lambda x: "Pass" if x >= 85 else "Fail")
print(df)
```
Removing Columns
Removing columns deletes unnecessary or irrelevant features. It helps reduce noise in ML models
Practical
```
df.drop("Country", axis=1, inplace=True)
print(df)
```
- axis=1 : column
- inplace=True :permanent change
Sorting Data
- sort_values():Sorts data based on column values.
Practical
```
print(df.sort_values(by="Marks", ascending=False))
```
- sort_index():Sorts data based on index.
Practical
```
print(df.sort_index(ascending=False))
```
Applying Functions
(apply(), map(), applymap())
- apply():Applies function row-wise or column-wise.
Practical
```
df["Marks"] = df["Marks"].apply(lambda x: x + 5)
print(df)
```
- map():Used on Series only for value mapping.
Practical
```
grade_map = {90: "A", 95: "A+", 93: "A", 97: "A+"}
df["Grade"] = df["Marks"].map(grade_map)
print(df)
```
- applymap():Applies function to each element of DataFrame.
Practical
```
print(df[["Age", "Marks"]].applymap(lambda x: x * 2))
```
Conditional Columns
Conditional columns are created using conditions (if-else).
Practical
```
df["Scholarship"] = df["Marks"].apply(
lambda x: "Yes" if x >= 90 else "No"
)
print(df)
```
Data Transformation
Data transformation converts data into new meaningful forms.
Practical
```
df["Marks_Percentage"] = (df["Marks"] / 100) * 100
print(df)
```
Used in normalization, scaling
Column Reordering
Rearranging column order for better readability.
Practical
```
df = df[["Name", "City", "Age", "Marks", "Scholarship"]]
print(df)
```
More Python tutorials
- What is Python and Why is it used for Data Science and Data Engineering?
- How Does Python Work in the Backend? Internal Working of Python
- Top 30 Python Interview Questions for Data Science
- 3-Month Python Roadmap to Excel in Data Science and Machine Learning
- Python Data Types Explained – A Beginner’s Guide
- test
All tutorials · Try the free PySpark compiler · Practice challenges