PySpark withColumn(): add or transform a column

PySpark withColumn(name, expr) returns a new DataFrame with a column added or replaced. The second argument is a Column expression built from col(), arithmetic, when()/otherwise(), or functions from pyspark.sql.functions. If the column name already exists it is overwritten; otherwise it is appended.

Add vs replace

withColumn("new", ...) appends a column; withColumn("existing", ...) replaces it. To only rename, use withColumnRenamed("old", "new").

Conditional columns

Use when()/otherwise() for CASE-style logic: withColumn("tier", when(col("amount") > 1000, "high").otherwise("low")).

withColumn, withColumnRenamed and withColumns

withColumn(name, expr) returns a new DataFrame with a column added, or replaced if the name already exists — which is the idiomatic way to overwrite a column in place. Use withColumnRenamed(old, new) to rename without touching the values. Since Spark 3.3 you can add several columns in one call with withColumns({'a': expr1, 'b': expr2}), which is both shorter and cheaper than chaining. Remember that DataFrames are immutable: withColumn never mutates the original, so you must assign the result back to a variable.

The chaining trap that slows jobs down

Use withColumn() for a small number of dependent transformations and select() or withColumns() for independent expressions. Repeated projections can make a large logical plan. For clarity across Spark versions, place an expression that depends on a newly created column in a subsequent transformation.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, lit, when

spark = SparkSession.builder.getOrCreate()
orders = spark.createDataFrame([(1, "120"), (2, "90")], ["id", "amount"])
result = (orders.withColumn("amount", col("amount").cast("int"))
    .withColumn("currency", lit("INR"))
    .withColumn("tier", when(col("amount") >= 100, "large").otherwise("small")))
for row in result.orderBy("id").collect():
    print(row.id, row.amount, row.currency, row.tier)

Replaces the string amount with an integer, adds a literal currency and derives a conditional tier. Separate transformations make the dependency on the converted amount clear.

Run this example in the free online PySpark compiler

Expected output

1 120 INR large
2 90 INR small

Frequently asked questions

Does PySpark withColumn modify the DataFrame in place?

No. DataFrames are immutable; withColumn returns a new DataFrame with the column added or replaced.

How do I rename a column in PySpark?

Use withColumnRenamed("old_name", "new_name"); withColumn is for computing values, not renaming.

How do I add a conditional column in PySpark?

Use when()/otherwise() from pyspark.sql.functions inside withColumn, e.g. when(col("x") > 0, "pos").otherwise("neg").

Practice challenges

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs