PySpark fillna(): replace null values
PySpark fillna(value) (alias df.na.fill(value)) replaces null values in a DataFrame. Pass a single value to fill all compatible columns, or a dict to fill specific columns with different values. Use dropna() to remove rows containing nulls instead of filling them.
Fill per column
df.fillna({"age": 0, "city": "unknown"}) assigns a default per named column. A scalar such as df.fillna(0, subset=["age"]) limits filling to compatible columns in the subset. With a dictionary, the dictionary keys choose the columns and the subset argument is ignored.
Drop vs fill vs coalesce
dropna() removes rows with nulls; fillna() substitutes a value; coalesce(col1, col2) returns the first non-null across columns.
fillna, na.fill and targeting specific columns
df.fillna(value) and df.na.fill(value) are the same operation. Passing a bare value fills every column of a matching type, which is rarely what you want. Two safer forms exist: restrict with a subset, as in df.fillna(0, subset=['amount', 'qty']), or pass a dictionary that maps each column to its own replacement, df.fillna({'amount': 0, 'country': 'unknown'}). The dictionary form is usually the clearest because the intent is visible per column.
Type matching and values that are not really null
For scalar fills, columns with incompatible types are ignored: numeric zero does not fill a string column. A per-column dictionary makes the intended replacements explicit. Empty strings and literal text such as "NULL" or "N/A" are not SQL nulls, so fillna does not replace those strings. Numeric fill also replaces floating-point NaN; the example below demonstrates both NaN and null in a double column. Choose defaults based on meaning: a missing age is not necessarily zero. For statistical imputation, compute the statistic separately or evaluate Spark ML Imputer.
| Input | fillna behavior | Decision |
|---|---|---|
| SQL null / Python None | Replaced by a compatible default | Choose a meaningful per-column value |
| Floating-point NaN | Replaced by a numeric fill | Check whether a constant or imputation is appropriate |
| Empty string or text "NULL" | Preserved as text | Normalize explicitly if it means missing |
| A row with missing fields | Preserved by fillna | Use dropna only when removing rows is intended |
Example (PySpark)
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
(1, None, "Delhi", float("nan")),
(2, 30, None, None),
(3, None, "", 2.5),
], "id INT, age INT, city STRING, score DOUBLE")
filled = df.fillna({"age": 0, "city": "unknown", "score": 0.0})
for row in filled.orderBy("id").collect():
print(row.id, row.age, repr(row.city), row.score)Null age and city values receive their named defaults. Both NaN and null scores become 0.0, while the empty city string remains empty. repr() makes that empty string visible. These defaults demonstrate behavior; choose domain-appropriate replacements for production data.
Run this example in the free online PySpark compiler
Expected output
1 0 'Delhi' 0.0
2 30 'unknown' 0.0
3 0 '' 2.5Frequently asked questions
How do I replace null values in PySpark?
Use df.fillna(value) or df.na.fill(value). Pass a dict to fill specific columns with different defaults.
What is the difference between fillna and dropna in PySpark?
fillna substitutes a value for nulls; dropna removes rows that contain nulls entirely.
How do I get the first non-null value across columns?
Use coalesce(col1, col2, ...) from pyspark.sql.functions, which returns the first non-null value per row.
Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs