PySpark join(): combine two DataFrames
PySpark `join()` combines two DataFrames on one or more key columns. You control the join type with the `how` argument — `inner` (default), `left`, `right`, `outer`, `left_semi`, `left_anti`. For a small lookup table, wrap it in `broadcast()` to avoid a shuffle and speed the join up.
Join types
`inner` keeps matching rows from both sides; `left` keeps all left rows; `outer` keeps everything; `left_anti` keeps left rows with no match. Choose based on whether you must retain unmatched rows.
Broadcast join for performance
When one DataFrame is small, `from pyspark.sql.functions import broadcast` and `large.join(broadcast(small), "id")` sends the small side to every executor and skips the expensive shuffle.
Example (PySpark)
from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast
spark = SparkSession.builder.getOrCreate()
orders = spark.createDataFrame([(1, 101), (2, 102), (3, 101)], ["order_id", "customer_id"])
customers = spark.createDataFrame([(101, "Alice"), (102, "Bob")], ["customer_id", "name"])
# inner join, broadcasting the small customers table
result = orders.join(broadcast(customers), "customer_id", "inner")
result.show()Joins each order to its customer name; broadcasting the small customers table avoids a shuffle.
Run this example in the free online PySpark compiler
Frequently asked questions
What is the default join type in PySpark?
inner. Rows are kept only when the join key exists in both DataFrames.
How do I do a broadcast join in PySpark?
Wrap the small DataFrame in broadcast(): large_df.join(broadcast(small_df), "key"). Spark sends the small side to every executor, avoiding a shuffle.
How do I join on multiple columns in PySpark?
Pass a list of column names, e.g. df1.join(df2, ["col1", "col2"], "inner").
How do I keep unmatched rows in a PySpark join?
Use how="left" to keep all left rows, or how="outer" to keep all rows from both DataFrames; unmatched columns become null.
Practice challenges
Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs