PySpark filter() and where(): keep rows that match a condition

PySpark `filter()` (and its alias `where()`) returns a new DataFrame keeping only the rows that satisfy a Boolean condition. You build the condition with `col()` comparisons and combine several with `&` (and), `|` (or) and `~` (not) — each condition must be wrapped in parentheses.

Combining conditions

Use & | ~ with parentheses around each comparison: df.filter((col("age") > 25) & (col("city") == "Delhi")). Python's and/or do not work on Column objects.

Membership and null checks

col("status").isin("active", "trial") matches a set of values; col("email").isNull() / isNotNull() filter on missing values.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, 32, "Delhi"), (2, 24, "Mumbai"), (3, 41, "Delhi")], ["id", "age", "city"])

df.filter((col("age") > 25) & (col("city") == "Delhi")).show()

Keeps only rows where age is over 25 and city is Delhi.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between filter() and where() in PySpark?

None — where() is an alias of filter(). They behave identically; use whichever reads better.

How do I filter with multiple conditions in PySpark?

Combine conditions with & (and), | (or), ~ (not), wrapping each comparison in parentheses, e.g. (col("a") > 1) & (col("b") == 2).

Why can't I use Python's 'and'/'or' in a PySpark filter?

Column objects don't support Python boolean operators; use the bitwise &, | and ~ with parentheses instead.

Practice challenges

• Fix the Broken Pipeline

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs