PySpark withColumn(): add or transform a column

PySpark `withColumn(name, expr)` returns a new DataFrame with a column added or replaced. The second argument is a Column expression built from `col()`, arithmetic, `when()/otherwise()`, or functions from `pyspark.sql.functions`. If the column name already exists it is overwritten; otherwise it is appended.

Add vs replace

withColumn("new", ...) appends a column; withColumn("existing", ...) replaces it. To only rename, use withColumnRenamed("old", "new").

Conditional columns

Use when()/otherwise() for CASE-style logic: withColumn("tier", when(col("amount") > 1000, "high").otherwise("low")).

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, 250.0), (2, 1200.0)], ["id", "amount"])

df.withColumn("amount_with_tax", col("amount") * 1.18) \
  .withColumn("tier", when(col("amount") > 1000, "high").otherwise("low")) \
  .show()

Adds a tax-adjusted column and a conditional tier column using when()/otherwise().

Run this example in the free online PySpark compiler

Frequently asked questions

Does PySpark withColumn modify the DataFrame in place?

No. DataFrames are immutable; withColumn returns a new DataFrame with the column added or replaced.

How do I rename a column in PySpark?

Use withColumnRenamed("old_name", "new_name"); withColumn is for computing values, not renaming.

How do I add a conditional column in PySpark?

Use when()/otherwise() from pyspark.sql.functions inside withColumn, e.g. when(col("x") > 0, "pos").otherwise("neg").

Practice challenges

• Fix the Broken Pipeline
• Parse Apache Logs with Regex

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs