Data Inspection & Exploration

Python tutorial · PySpark.in

Data Inspection & Exploration

Data Inspection and Exploration is the process of understanding the structure, size, content, and statistical properties of a dataset before performing cleaning or modeling..

Why Do We Need Data Inspection?

✔ Understand dataset structure
✔ Detect missing values
✔ Check data types
✔ Find outliers
✔ Decide cleaning strategy

Always FIRST STEP in EDA(Exploratory Data Analysis)

Exploratory Data Analysis (EDA) is the process of analyzing, summarizing, and visualizing datasets to discover patterns, trends, relationships, and anomalies before building models.

Sample DataFrame Used

```

import pandas as pd

data = {

"Name": ["Megha", "Amit", "Riya", "Neha", "Rahul"],

"Age": [21, 22, 21, 23, 22],

"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Delhi"],

"Marks": [85, 90, 88, 92, 80]

}

df = pd.DataFrame(data)

print(df)

```

1. head() and tail()

head() is a Pandas function used to display the first few rows of a DataFrame.
tail() is a Pandas function used to display the last few rows of a DataFrame.

By default:

head() → first 5 rows
tail() → last 5 rows

Used to quickly inspect large datasets.

```

print(df.head())

```

print(df.tail())

```

2. shape, size, ndim

Shape: returns a tuple showing the number of rows and columns in a DataFrame.

```

print(df.shape)

```

Means: 5 rows, 4 columns

Size: returns the total number of values present in the DataFrame.

```

print(df.size)

```

Formula: rows × columns = 5 × 4 = 20

Ndim:returns the number of dimensions of the data structure.

```

print(df.ndim)

```

DataFrame is always 2-Dimensional

3. info()

info() provides a complete technical summary of the DataFrame including:

Column names
Non-null values
Data types
Memory usage

Used by data scientists first to understand dataset structure and cleanliness.

```

print(df.info())

```

4. describe()

describe() generates statistical summary of numerical columns.

It includes:

count
mean
standard deviation (std)
min
max
quartiles

```

print(df.describe())

```

Used in EDA (Exploratory Data Analysis)

5. value_counts()

value_counts() returns the frequency count of each unique value in a column.

```

print(df["City"].value_counts())

```

Used for categorical data analysis

6. unique() and nunique()

unique()

unique() returns all distinct values in a column.

```

print(df["City"].unique())

```

nunique()

nunique() returns the number of unique values.

```

print(df["City"].nunique())

```

ADVANCED (Interview Level)

Filtering

Filtering selects rows based on a condition.

```

print(df[df["Marks"] > 85])

```

Sorting

sort_values() sorts data in ascending or descending order.

```

print(df.sort_values(by="Marks", ascending=False))

```

Missing Values Handling

Checks missing (NaN) values.

isnull()

```

print(df.isnull())

```

Output: all False (no missing values)

fillna()

Fills missing values with a specified value.

```

print(df.fillna(0))

```

apply()

apply() applies a custom function to each value of a column.

```

df["Updated_Marks"] = df["Marks"].apply(lambda x: x + 5)

print(df)

```

Data Inspection & Exploration