Data Cleaning & Data Preprocessing
Python tutorial · PySpark.in
What is Data Cleaning?
Data Cleaning is the process of detecting and correcting incomplete, incorrect, duplicate, or inconsistent data in a dataset.
Clean data means Better ML results or good prediction and accuracy.
What is Data Preprocessing?
Data Preprocessing is the process of transforming cleaned data into a format suitable for machine learning algorithms.
- Cleaning data simply means fix data
- Preprocessing means prepare data
Difference Between Cleaning & Preprocessing
Data Cleaning | Data Preprocessing |
|---|---|
Remove errors | Transform data |
Handle missing values | Scaling / Encoding |
Remove duplicates | Feature engineering |
Fix data types | Train-test split |
Sample Dataset
```
import pandas as pd
data = {
"Name": ["Megha", "Amit", "Riya", "Neha", "Rahul", "Rahul"],
"Age": [21, 22, None, 21, 22, 22],
"City": ["Delhi", "Mumbai", "Delhi", "Pune", None, None],
"Marks": [85, 90, 88, None, 80, 80]
}
df = pd.DataFrame(data)
print(df)
```
PART 1: DATA CLEANING
1. Handling Missing Values
Missing values are empty or NaN values in data. ML models cannot handle NaN directly.
Detect Missing Values
```
print(df.isnull())
```
Count Missing Values
```
print(df.isnull().sum())
```
Remove Missing Values
```
X=df.dropna()
print(X)
```
Risk is the loss of the data.
Fill Missing Values
```
Y=df.fillna(0)
print(Y)
```
OR
```
T=df["Age"].fillna(df["Age"].mean(), inplace=True)
print(T)
```
2. Removing Duplicates
Duplicate rows can bias results.
```
X=df.drop_duplicates()
print(X)
```
3. Fixing Data Types
Wrong data types cause calculation errors.
```
S=df["Age"] = df["Age"].astype(float)
print(S)
```
4. Replacing Wrong Values
```
T=df["City"] = df["City"].replace("Delhi", "New Delhi")
print(T)
```
5. Renaming Columns
```
S=df.rename(columns={"Marks": "Score"}, inplace=True)
print(S)
```
PART 2: DATA PREPROCESSING
6. Encoding Categorical Data
ML models need numeric input.
Label Encoding
```
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df["City"] = le.fit_transform(df["City"])
print(le)
```
One Hot Encoding
```
X=pd.get_dummies(df, columns=["City"])
print(X)
```
7. Feature Scaling
Scaling ensures equal importance to all features.
Standardization
```
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df[["Age", "Score"]] = scaler.fit_transform(df[["Age", "Score"]])
print(scaler)
```
Normalization
```
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
df[["Age", "Score"]] = scaler.fit_transform(df[["Age", "Score"]])
print(scaler)
```
8. Outlier Handling
Outliers are extreme values.
```
Z=df.boxplot(column="Score")
print(Z)
```
9. Feature Selection
```
hide
X=df.corr()
print(X)
```
Remove less important features
Train-Test Split
```
from sklearn.model_selection import train_test_split
X = df.drop("Score", axis=1)
y = df["Score"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
print(X)
print(y)
```
More Python tutorials
- What is Python and Why is it used for Data Science and Data Engineering?
- How Does Python Work in the Backend? Internal Working of Python
- Top 30 Python Interview Questions for Data Science
- 3-Month Python Roadmap to Excel in Data Science and Machine Learning
- Python Data Types Explained – A Beginner’s Guide
- test
All tutorials · Try the free PySpark compiler · Practice challenges