Data Selection & Indexing

Python tutorial · PySpark.in

What is Data Selection & Indexing?

Data Selection and Indexing in Pandas is the process of accessing, extracting, and manipulating specific rows and columns from a DataFrame using labels, positions, or conditions.

In simple words, it allows us to pick required data from a large dataset.

What is Indexing?

Indexing refers to assigning labels (index values) to rows in a DataFrame so that data can be accessed quickly and accurately.

Example

Think of:

• Book index page → page number
• DataFrame index → row number

Index is the address of the data

Why do we need Data Selection & Indexing?

1. To Handle Large Datasets

Real-world datasets has lakhs of rows.

Example:
We don’t want full data, we want only:

• students with marks > 85
• people from Delhi

Selection helps extract only required data

2. To Improve Performance

Selecting specific rows/columns makes operations faster and efficient.

Example:
Using index instead of scanning full dataset saves time.

3. To Perform Data Analysis (EDA)

In EDA we handle data by:

• filter
• compare
• inspect

all the above are impossible without using selection & indexing

4.To Prepare Data for Machine Learning

ML models: needs two things mainly:

• input features (X)
• target variable (y)

```

import pandas as pd

data = {

"Name": ["Megha", "Amit", "Riya", "Neha", "Rahul"],

"Age": [21, 22, 21, 23, 22],

"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Delhi"],

"Marks": [85, 90, 88, 92, 80]

}

df = pd.DataFrame(data)

print(df)

```

Example:

```

X = df[["Age", "Marks"]]

y = df["Result"]

print(X)

print(y)

```

This is column selection

5.To Apply Conditions on Data

Example:

```

X=df[df["Marks"] > 85]

print(X)

```

This is boolean indexing

6.To Read and Modify Data Correctly

Using loc and iloc avoids wrong data access errors.

Real-Life Example

Imagine an Excel Sheet with 1,00,000 students

Without selection:

• Data confusing
• Slow processing

With selection:

• Only toppers
• Only one city
• Only one subject

That’s Data Selection & Indexing

Where is it Used?

• Data Analysis
• Data Cleaning
• Machine Learning
• Deep Learning
• Business Analytics
  

Sample DataFrame

```

import pandas as pd

data = {

"Name": ["Megha", "Amit", "Riya", "Neha", "Rahul"],

"Age": [21, 22, 21, 23, 22],

"City": ["Delhi", "Mumbai", "Delhi", "Pune", "Delhi"],

"Marks": [85, 90, 88, 92, 80]

}

df = pd.DataFrame(data)

print(df)

```

Column Selection

Column Selection means selecting one or more columns from a DataFrame.

Columns can be selected using:

• column name
• list of column names

```

print(df["Name"])

```

```

print(df[["Name", "Marks"]])

```

Use: Feature selection in ML

Row Selection

Row selection means selecting specific rows from a DataFrame.

```

print(df[0:3])

```

Uses index slicing

loc[] (Label-based Indexing)

loc[] is used to select rows and columns by label name.

• Includes last index
• Uses row labels & column names

```

print(df.loc[1])

```

```

print(df.loc[0:2, ["Name", "Marks"]])

```

iloc[] (Position-based Indexing)

iloc[] is used to select rows and columns by index position.

• Starts from 0
• Excludes last index

```

print(df.iloc[0])

```

```

print(df.iloc[0:3, 0:2])

```

Key Differences in loc[] and iloc[]

Boolean Indexing

Boolean Indexing selects rows based on True / False conditions.

```

print(df[df["Marks"] > 85])

```

Used in data filtering

Filtering Rows

Filtering is selecting rows that satisfy given conditions.

```

print(df[(df["City"] == "Delhi") & (df["Marks"] > 85)])

```

Used in EDA & ML pipelines

Setting Index

set_index() changes an existing column into the index of DataFrame.

```

df_indexed = df.set_index("Name")

print(df_indexed)

```

Improves data lookup speed

Resetting Index

reset_index() resets index back to default numeric index.

```

print(df_indexed.reset_index())

```

More Python tutorials

• What is Python and Why is it used for Data Science and Data Engineering?
• How Does Python Work in the Backend? Internal Working of Python
• Top 30 Python Interview Questions for Data Science
• 3-Month Python Roadmap to Excel in Data Science and Machine Learning
• Python Data Types Explained – A Beginner’s Guide
• test

All tutorials · Try the free PySpark compiler · Practice challenges