PySpark select(): pick and compute columns

PySpark `select()` returns a new DataFrame containing only the columns (or column expressions) you name. You can pass column names as strings, `col()` objects, or expressions built with functions and `.alias()` to rename the output. Use `selectExpr()` to write the expressions as SQL strings.

Select vs withColumn

select() projects a specific set of columns (dropping the rest); withColumn() adds/replaces one column while keeping all others. Use select when you want to reshape the whole row.

selectExpr for SQL expressions

df.selectExpr("id", "amount * 1.18 AS with_tax") lets you write SQL-style expressions instead of building Column objects.

select, selectExpr and referring to columns

select() takes column names as strings or Column objects, so df.select('id', F.col('name'), (F.col('price') * 1.2).alias('gross')) is all valid in one call. selectExpr() takes SQL expression strings instead, which is often more compact: df.selectExpr('id', 'price * 1.2 as gross'). Use alias() (or 'as' in selectExpr) to name derived columns — otherwise you end up with machine-generated names that are awkward to reference later. To select every column plus a new one, use df.select('*', expr.alias('new')).

Ambiguous columns and nested fields

After a join, two columns can share a name and select('id') will fail as ambiguous. Qualify it by referencing the source DataFrame — df1['id'] or df1.id — or rename before joining. For nested data, dot notation reaches into structs: df.select('address.city'). To flatten every field of a struct at once, use df.select('address.*'). Selecting only the columns you actually need early in the pipeline is also a genuine optimisation for columnar formats such as Parquet, because Spark can skip reading the rest from disk.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, "Alice", 250.0)], ["id", "name", "amount"])

df.select(col("name"), (col("amount") * 1.18).alias("amount_with_tax")).show()

Projects the name column and a computed, aliased tax-adjusted amount.

Run this example in the free online PySpark compiler

Frequently asked questions

What does PySpark select() do?

It returns a new DataFrame containing only the columns or expressions you pass, dropping all others.

How do I rename a column inside select()?

Use .alias(): df.select(col("old").alias("new")). For a plain rename without select, use withColumnRenamed().

What is selectExpr in PySpark?

A variant of select that accepts SQL expression strings, e.g. df.selectExpr("id", "price * qty AS total").

Practice challenges

• Calculate Total Sales and Average Order Value by Customer

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs