PySpark filter() and where(): keep rows that match a condition

PySpark `filter()` (and its alias `where()`) returns a new DataFrame keeping only the rows that satisfy a Boolean condition. You build the condition with `col()` comparisons and combine several with `&` (and), `|` (or) and `~` (not) — each condition must be wrapped in parentheses.

Combining conditions

Use & | ~ with parentheses around each comparison: df.filter((col("age") > 25) & (col("city") == "Delhi")). Python's and/or do not work on Column objects.

Membership and null checks

col("status").isin("active", "trial") matches a set of values; col("email").isNull() / isNotNull() filter on missing values.

filter and where, and combining conditions correctly

filter() and where() are the same function; where() exists so SQL users feel at home. You can pass a Column expression, df.filter(F.col('age') > 30), or a SQL string, df.filter('age > 30'). The classic mistake is combining conditions with Python's and/or keywords, which raises an error because Column objects do not implement them. Use the bitwise operators & (and), | (or) and ~ (not), and wrap every condition in parentheses: df.filter((F.col('age') > 30) & (F.col('country') == 'IN')). The parentheses are mandatory because & binds tighter than the comparison operators.

Nulls, membership tests and pushdown

Comparisons against null return null, not false, so a row where the column is null is dropped by both col == 'x' and col != 'x'. Test for missing values explicitly with isNull() or isNotNull(), and use eqNullSafe() when null should count as a match. For membership use isin(['a','b']) instead of chaining ors, and between(lo, hi) for inclusive ranges. Filtering early pays off: with Parquet or Delta, predicates on partition and statistics columns are pushed down to the scan, so Spark skips entire files rather than reading and discarding rows.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, 32, "Delhi"), (2, 24, "Mumbai"), (3, 41, "Delhi")], ["id", "age", "city"])

df.filter((col("age") > 25) & (col("city") == "Delhi")).show()

Keeps only rows where age is over 25 and city is Delhi.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between filter() and where() in PySpark?

None — where() is an alias of filter(). They behave identically; use whichever reads better.

How do I filter with multiple conditions in PySpark?

Combine conditions with & (and), | (or), ~ (not), wrapping each comparison in parentheses, e.g. (col("a") > 1) & (col("b") == 2).

Why can't I use Python's 'and'/'or' in a PySpark filter?

Column objects don't support Python boolean operators; use the bitwise &, | and ~ with parentheses instead.

Practice challenges

• Fix the Broken Pipeline

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs