PySpark orderBy() and sort(): order rows

PySpark `orderBy()` (alias `sort()`) returns a new DataFrame with rows sorted by one or more columns. By default sorting is ascending; use `col("x").desc()` or `.asc()` to control direction per column, and pass several columns to break ties.

Sort direction

df.orderBy(col("amount").desc()) sorts high-to-low. Mix directions across columns: orderBy(col("region").asc(), col("amount").desc()).

Nulls ordering

Use desc_nulls_last() / asc_nulls_first() when you need explicit control over where null values appear in the sort.

orderBy vs sort, and controlling null placement

In PySpark, orderBy() and sort() are aliases for the same operation, so pick whichever reads better. You control direction per column with F.col('x').desc() or the asc()/desc() helpers, and you can sort by several columns at once: df.orderBy(F.col('dept').asc(), F.col('salary').desc()). Null handling is where people get caught out. By default nulls sort first ascending and last descending, which quietly changes your 'top N' results when a column is nullable. Make it explicit with desc_nulls_last() or asc_nulls_first() rather than relying on the default.

Why sorting is expensive, and how to avoid a full sort

A global orderBy forces a full shuffle: Spark must range-partition the data so that every partition holds a contiguous slice of the sort key. On large datasets that is one of the most expensive operations you can run. If you only need the top few rows, df.orderBy(...).limit(n) lets Spark use a much cheaper top-N operator instead of sorting everything. If you only need order within groups, use a window function partitioned by the group key rather than sorting the whole DataFrame. And if you are sorting purely to inspect data, sort after filtering, not before.

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([("North", 300), ("South", 500), ("North", 100)], ["region", "amount"])

df.orderBy(col("region").asc(), col("amount").desc()).show()

Sorts by region ascending, then amount descending within each region.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between sort() and orderBy() in PySpark?

They are aliases and behave the same; both return a globally sorted DataFrame.

How do I sort descending in PySpark?

Use col("column").desc() inside orderBy, e.g. df.orderBy(col("amount").desc()).

How do I sort by multiple columns in PySpark?

Pass several column expressions to orderBy; earlier columns take priority and later ones break ties.

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs