PySpark fillna(): replace null values

PySpark `fillna(value)` (alias `df.na.fill(value)`) replaces null values in a DataFrame. Pass a single value to fill all compatible columns, or a dict to fill specific columns with different values. Use `dropna()` to remove rows containing nulls instead of filling them.

Fill per column

df.fillna({"age": 0, "city": "unknown"}) fills each named column with its own default; a single value like fillna(0) fills all numeric columns.

Drop vs fill vs coalesce

dropna() removes rows with nulls; fillna() substitutes a value; coalesce(col1, col2) returns the first non-null across columns.

fillna, na.fill and targeting specific columns

df.fillna(value) and df.na.fill(value) are the same operation. Passing a bare value fills every column of a matching type, which is rarely what you want. Two safer forms exist: restrict with a subset, as in df.fillna(0, subset=['amount', 'qty']), or pass a dictionary that maps each column to its own replacement, df.fillna({'amount': 0, 'country': 'unknown'}). The dictionary form is usually the clearest because the intent is visible per column.

Type matching and values that are not really null

fillna silently ignores columns whose type does not match the value you supply — filling with 0 will not touch a string column, and filling with a string will not touch a numeric one. That is the most common reason a fillna appears to do nothing. Also remember that empty strings, the literal text 'NULL' or 'N/A', and NaN in a double column are not the same as SQL NULL. Normalise those first (with when/otherwise or NULLIF-style logic) so every missing value is a real null, then fill once. For imputing a statistic rather than a constant, compute it separately or use the Imputer from spark.ml.

Example (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, None, "Delhi"), (2, 30, None)], ["id", "age", "city"])

df.fillna({"age": 0, "city": "unknown"}).show()

Fills null age with 0 and null city with 'unknown' using a per-column dict.

Run this example in the free online PySpark compiler

Frequently asked questions

How do I replace null values in PySpark?

Use df.fillna(value) or df.na.fill(value). Pass a dict to fill specific columns with different defaults.

What is the difference between fillna and dropna in PySpark?

fillna substitutes a value for nulls; dropna removes rows that contain nulls entirely.

How do I get the first non-null value across columns?

Use coalesce(col1, col2, ...) from pyspark.sql.functions, which returns the first non-null value per row.

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs