PySpark select(): pick and compute columns
PySpark select() returns a new DataFrame containing only the columns (or column expressions) you name. You can pass column names as strings, col() objects, or expressions built with functions and .alias() to rename the output. Use selectExpr() to write the expressions as SQL strings.
Select vs withColumn
select() projects a specific set of columns (dropping the rest); withColumn() adds/replaces one column while keeping all others. Use select when you want to reshape the whole row.
selectExpr for SQL expressions
df.selectExpr("id", "amount * 1.18 AS with_tax") lets you write SQL-style expressions instead of building Column objects.
select, selectExpr and referring to columns
select() takes column names as strings or Column objects, so df.select('id', F.col('name'), (F.col('price') * 1.2).alias('gross')) is all valid in one call. selectExpr() takes SQL expression strings instead, which is often more compact: df.selectExpr('id', 'price * 1.2 as gross'). Use alias() (or 'as' in selectExpr) to name derived columns — otherwise you end up with machine-generated names that are awkward to reference later. To select every column plus a new one, use df.select('*', expr.alias('new')).
Ambiguous columns and nested fields
After a join, two columns can share a name and select('id') will fail as ambiguous. Qualify it by referencing the source DataFrame — df1['id'] or df1.id — or rename before joining. For nested data, dot notation reaches into structs: df.select('address.city'). To flatten every field of a struct at once, use df.select('address.*'). Selecting only the columns you actually need early in the pipeline is also a genuine optimisation for columnar formats such as Parquet, because Spark can skip reading the rest from disk.
Example (PySpark)
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, "Alice", 250.0)], ["id", "name", "amount"])
df.select(col("name"), (col("amount") * 1.18).alias("amount_with_tax")).show()Projects the name column and a computed, aliased tax-adjusted amount.
Frequently asked questions
What does PySpark select() do?
It returns a new DataFrame containing only the columns or expressions you pass, dropping all others.
How do I rename a column inside select()?
Use .alias(): df.select(col("old").alias("new")). For a plain rename without select, use withColumnRenamed().
What is selectExpr in PySpark?
A variant of select that accepts SQL expression strings, e.g. df.selectExpr("id", "price * qty AS total").
Practice challenges
Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs