PySpark select(): pick and compute columns

PySpark `select()` returns a new DataFrame containing only the columns (or column expressions) you name. You can pass column names as strings, `col()` objects, or expressions built with functions and `.alias()` to rename the output. Use `selectExpr()` to write the expressions as SQL strings.

Select vs withColumn

select() projects a specific set of columns (dropping the rest); withColumn() adds/replaces one column while keeping all others. Use select when you want to reshape the whole row.

selectExpr for SQL expressions

df.selectExpr("id", "amount * 1.18 AS with_tax") lets you write SQL-style expressions instead of building Column objects.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, "Alice", 250.0)], ["id", "name", "amount"])

df.select(col("name"), (col("amount") * 1.18).alias("amount_with_tax")).show()

Projects the name column and a computed, aliased tax-adjusted amount.

Run this example in the free online PySpark compiler

Frequently asked questions

What does PySpark select() do?

It returns a new DataFrame containing only the columns or expressions you pass, dropping all others.

How do I rename a column inside select()?

Use .alias(): df.select(col("old").alias("new")). For a plain rename without select, use withColumnRenamed().

What is selectExpr in PySpark?

A variant of select that accepts SQL expression strings, e.g. df.selectExpr("id", "price * qty AS total").

Practice challenges

• Calculate Total Sales and Average Order Value by Customer

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs