PySpark withColumn(): add or transform a column

PySpark `withColumn(name, expr)` returns a new DataFrame with a column added or replaced. The second argument is a Column expression built from `col()`, arithmetic, `when()/otherwise()`, or functions from `pyspark.sql.functions`. If the column name already exists it is overwritten; otherwise it is appended.

Add vs replace

withColumn("new", ...) appends a column; withColumn("existing", ...) replaces it. To only rename, use withColumnRenamed("old", "new").

Conditional columns

Use when()/otherwise() for CASE-style logic: withColumn("tier", when(col("amount") > 1000, "high").otherwise("low")).

withColumn, withColumnRenamed and withColumns

withColumn(name, expr) returns a new DataFrame with a column added, or replaced if the name already exists — which is the idiomatic way to overwrite a column in place. Use withColumnRenamed(old, new) to rename without touching the values. Since Spark 3.3 you can add several columns in one call with withColumns({'a': expr1, 'b': expr2}), which is both shorter and cheaper than chaining. Remember that DataFrames are immutable: withColumn never mutates the original, so you must assign the result back to a variable.

The chaining trap that slows jobs down

Calling withColumn dozens of times in a loop is a well-known performance problem. Each call adds another projection to the logical plan, and with enough of them the Catalyst analyzer slows to a crawl — plan analysis time can grow quadratically. If you are adding many columns, build a list of expressions and apply them in a single select(*exprs), or use withColumns with a dict. Another common mistake is referencing a column you created earlier in the same select: that works with chained withColumn calls but not inside one select, where all expressions are resolved against the input schema.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, 250.0), (2, 1200.0)], ["id", "amount"])

df.withColumn("amount_with_tax", col("amount") * 1.18) \
  .withColumn("tier", when(col("amount") > 1000, "high").otherwise("low")) \
  .show()

Adds a tax-adjusted column and a conditional tier column using when()/otherwise().

Run this example in the free online PySpark compiler

Frequently asked questions

Does PySpark withColumn modify the DataFrame in place?

No. DataFrames are immutable; withColumn returns a new DataFrame with the column added or replaced.

How do I rename a column in PySpark?

Use withColumnRenamed("old_name", "new_name"); withColumn is for computing values, not renaming.

How do I add a conditional column in PySpark?

Use when()/otherwise() from pyspark.sql.functions inside withColumn, e.g. when(col("x") > 0, "pos").otherwise("neg").

Practice challenges

• Fix the Broken Pipeline
• Parse Apache Logs with Regex

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs