Dispalying DataFrames

Spark tutorial · PySpark.in

Displaying Data in PySpark

When you first load a DataFrame in PySpark, you need ways to inspect it without loading an entire distributed dataset into memory. Unlike Pandas (which prints a full table), PySpark provides optimized methods that show a sample of the data or return small subsets as Python objects. This section covers the most common techniques— show(), take(), and collect()—along with best practices and use cases.

1. show() – The Most Common Display Method

The show() method prints rows in a clean, tabular format. By default it shows the first 20 rows and truncates long string columns to 20 characters. You can adjust the number of rows and truncation behavior:

• Basic usage: df.show() displays the top 20 rows with truncated columns.
• Display more rows: Pass an integer to show n rows: df.show(5).
• Disable truncation: Pass truncate=False to show full column values: df.show(truncate=False).
• Combine options: df.show(n, truncate=False) shows n rows with full values.
• Vertical display: Set vertical=True to print each row in a column-by-column format.

Example 1: Display full DataFrame

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ShowExample").getOrCreate()

data = [(1, "Laptop", 55000, 2), (2, "Mouse", 500, 10), (3, "Keyboard", 1200, 4)]

columns = ["id", "product", "price", "quantity"]

df = spark.createDataFrame(data, columns)

df.show() # default: 20 rows, truncated

df.show(5) # show up to 5 rows

df.show(truncate=False) # show full column values

df.show(5, truncate=False) # show 5 rows with no truncation

df.show(vertical=True) # print each row vertically

```

2. take() – Returns Rows as Python Objects

Use take(n) to retrieve the first n rows as a list of Row objects. According to the API documentation, DataFrame.take(num) returns the first num rows as a list of Row. This is especially useful for programmatic checks, unit tests, or conditional logic where you need direct access to row contents.

Example:

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("TakeExample") \

.master("local[*]") \

.getOrCreate()

df = spark.createDataFrame(

[(1,"Apple"),(2,"Banana"),(3,"Grapes")],

["id","fruit"]

)

rows = df.take(2)

for r in rows:

print(r)

```

When to use?

• When building conditional logic
• When preparing unit tests
• When validating ETL first few rows

3. collect() – Bring Entire Dataset to Driver

The collect() method returns all records as a list of Row objects. The documentation cautions that this should only be used when the resulting list is small, because it loads the entire DataFrame into the driver’s memory. Avoid calling collect() on large datasets, as it can lead to memory errors or crash the driver.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("CollectExample") \

.master("local[*]") \

.getOrCreate()

df = spark.createDataFrame(

[(10,"X"),(20,"Y"),(30,"Z")],

["score","grade"]

)

all_rows = df.collect()

print(all_rows)

```

Avoid collect() on large datasets

It can cause:

• Memory overflow
• Driver crash
• OutOfMemoryError

Best Practices

• Use show() for quick previews. Adjust the number of rows and truncation as needed.
• Use take() for programmatic inspection. It’s explicit and returns Python objects.
• Reserve collect() for small or aggregated data. Avoid on large DataFrames; consider show() or take() instead.
• Combine with filters and projections to narrow down data before collecting.

Comparison Table:

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges