RDD vs DataFrame vs Dataset in Spark: Differences

Spark tutorial · PySpark.in

RDD vs DataFrame: an RDD is a distributed collection of Python or JVM objects that Spark cannot look inside, while a DataFrame is a distributed table with named, typed columns that Spark optimises for you. In PySpark, use DataFrames for almost everything; this guide shows why, with runnable examples, and when an RDD still makes sense.

Run the examples: Every code block is self-contained. Press Run on any block; this page runs real Spark (version 3.5) in local mode.

RDD vs DataFrame vs Dataset: the quick answer

RDDDataFrameDataset
What it isDistributed collection of objectsDistributed table of rows with named columnsTyped, distributed collection of objects with a schema
SchemaNone: Spark sees opaque objectsYes: column names and typesYes, from a Scala case class or Java bean
Optimised by CatalystNo: runs your functions as writtenYes: query planning, filter and column pruningYes
Memory formatSerialised objects (Python pickles in PySpark)Tungsten binary rows, compact and fastTungsten binary rows
Type safetyIn Scala at compile time; none in PythonChecked when the plan is analysed (run time)Compile time (Scala/Java)
LanguagesScala, Java, PythonScala, Java, Python, R, SQLScala and Java only
Speed in PySparkSlowest: data moves between the JVM and PythonFast: work stays in the JVMNot available in Python
Use it forRare low-level work and legacy codeAlmost every PySpark jobTyped pipelines in Scala/Java

In Scala, DataFrame is simply Dataset[Row]: since Spark 2.0 they are one API. Python has no compile-time types, so PySpark only offers RDDs and DataFrames.

What is an RDD?

An RDD (Resilient Distributed Dataset) is Spark's original abstraction: an immutable collection split into partitions across the cluster. You transform it with functions such as map, filter and reduceByKey, and Spark can rebuild a lost partition from its lineage. Spark does not know what is inside each element, so it cannot optimise your code; it runs your functions exactly as written. For the full introduction, see Understanding Apache Spark RDDs.

What is a DataFrame?

A DataFrame is a distributed table: rows with named, typed columns, like a SQL table or a pandas DataFrame spread across a cluster. Because Spark knows the schema and you describe what you want (select, filter, group) rather than how, the Catalyst optimizer can reorder and prune the work, and the Tungsten engine stores rows in a compact binary format and generates fast code for them.

What is a Dataset?

A Dataset adds compile-time type safety on top of a DataFrame: in Scala you work with your own case class, and a typo in a field name fails at compile time instead of at run time. It exists only in Scala and Java. For PySpark developers the practical choice is RDD or DataFrame, and the answer is nearly always DataFrame.

The same task with an RDD and with a DataFrame

Revenue per city, first with the RDD API. Note the lambdas: each one is a Python function that Spark must run in Python worker processes.

```python

from pyspark.sql import SparkSession

 

spark = SparkSession.builder.appName("RddRevenue").getOrCreate()

sc = spark.sparkContext

 

sales = [("Delhi", 900.0), ("Pune", 300.0), ("Delhi", 250.0),

("Jaipur", 1100.0), ("Pune", 400.0), ("Delhi", 380.0)]

 

rdd = sc.parallelize(sales)

revenue = (rdd.filter(lambda row: row[1] > 280)

.reduceByKey(lambda a, b: a + b)

.sortByKey())

 

print(revenue.collect())

```

Now the same result with the DataFrame API. There are no Python functions to call per row: every step is a built-in expression that runs inside the JVM, and explain() shows the optimised plan Spark built for it.

```python

from pyspark.sql import SparkSession, functions as F

 

spark = SparkSession.builder.appName("DataFrameRevenue").getOrCreate()

 

sales = spark.createDataFrame(

[("Delhi", 900.0), ("Pune", 300.0), ("Delhi", 250.0),

("Jaipur", 1100.0), ("Pune", 400.0), ("Delhi", 380.0)],

["city", "amount"],

)

 

revenue = (sales.filter(F.col("amount") > 280)

.groupBy("city")

.agg(F.sum("amount").alias("revenue"))

.orderBy("city"))

 

revenue.show()

revenue.explain()

```

Both give Delhi 1280, Jaipur 1100 and Pune 700. The difference is what Spark can see: the DataFrame plan shows a Filter and a two-step HashAggregate (a partial sum before the shuffle and a final sum after it), which Spark chose by itself. With the RDD, Spark only knows "call this lambda".

Why DataFrames are faster in PySpark

  1. No Python round trip. RDD lambdas run in Python worker processes, so every row is pickled in the JVM, sent to Python, processed and sent back. DataFrame expressions such as F.col("amount") > 280 are translated into JVM code and never leave the JVM.
  2. Catalyst optimisation. Spark can push filters closer to the data, read only the columns you use, pick join strategies such as broadcast joins, and with Adaptive Query Execution adjust the plan at run time.
  3. Tungsten storage. DataFrame rows are kept in a compact binary format, so caching and shuffles move fewer bytes than serialised Python objects.
  4. Source pushdown. With Parquet, ORC, Delta or JDBC sources, filters and column selections can be handed to the reader, so less data is read at all.

Python UDFs: A Python UDF on a DataFrame brings back the Python round trip for that column. Prefer built-in functions, and if you need custom Python, a pandas UDF moves data in Arrow batches and is much faster than a row-by-row UDF.

Converting between RDDs and DataFrames

You can move between the two APIs at any time. df.rdd gives an RDD of Row objects, and toDF() or spark.createDataFrame() turns an RDD back into a DataFrame. Each conversion moves data through Python, so do it once, not inside a loop.

```python

from pyspark.sql import SparkSession

from pyspark.sql.types import StructType, StructField, StringType, DoubleType

 

spark = SparkSession.builder.appName("ConvertDemo").getOrCreate()

 

df = spark.createDataFrame([("Delhi", 900.0), ("Pune", 300.0)], ["city", "amount"])

 

rows = df.rdd.take(2)

print("DataFrame to RDD gives Row objects:", rows)

print("Fields by name:", rows[0]["city"], rows[0].amount)

 

rdd = spark.sparkContext.parallelize([("Jaipur", 1100.0), ("Agra", 450.0)])

df_quick = rdd.toDF(["city", "amount"])

df_quick.printSchema()

 

schema = StructType([StructField("city", StringType(), False),

StructField("amount", DoubleType(), False)])

df_typed = spark.createDataFrame(rdd, schema)

df_typed.show()

```

When should you still use an RDD?

Note that Spark Connect (Spark 3.4+), and the shared clusters on platforms that use it, does not support the RDD API at all: new code that needs to run there must use DataFrames. For per-partition work, mapInPandas and mapInArrow are the DataFrame alternatives to mapPartitions.

RDD vs DataFrame interview questions

What is the difference between an RDD and a DataFrame?

An RDD is a low-level distributed collection of objects with no schema, so Spark cannot optimise it. A DataFrame has named, typed columns, is optimised by Catalyst and stored in Tungsten's binary format, and is usually much faster, especially from Python.

Why is the RDD API slow in PySpark?

Each RDD function is a Python function, so rows are serialised from the JVM to Python worker processes and back. DataFrame operations stay in the JVM.

Is a DataFrame built on top of RDDs?

Yes. Spark still executes everything as RDDs of internal binary rows; the DataFrame API adds a schema and an optimiser on top. df.rdd exposes those rows as Python Row objects.

What is a Dataset, and can I use it in PySpark?

A Dataset is a typed DataFrame with compile-time checks, available in Scala and Java. PySpark has no Dataset API because Python has no compile-time types; in Scala a DataFrame is Dataset[Row].

Are RDDs and DataFrames lazy?

Both are. Transformations only build a plan; an action such as count() or collect() runs it. See lazy evaluation in PySpark.

Do RDDs and DataFrames give the same fault tolerance?

Yes. Both recover lost partitions from lineage, because a DataFrame runs on RDDs underneath.

Practise next

More Spark tutorials

PySpark tutorial: all lessons · All tutorials · Try the free PySpark compiler · Practice challenges