Exploring DataFrames

Spark tutorial · PySpark.in

Exploring Data Frames in PySpark

Exploring data is a crucial first step before performing transformations, joins, or analytics. Because PySpark runs on a cluster, the way you examine data differs from Pandas: operations are lazily evaluated and distributed across partitions.

1. Displaying Data ():

Displays the show([n], truncate=False) : Displays the first n rows in a tabular format (default is 20). Use truncate=False to show full column values.

Example

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.show() # Displays the first 20 rows.

df.show(50)# Show more rows:

df.show(truncate=False) # Show full column values:

```

1.2 head(): Returns the first row (or first n rows) as Python objects rather than a DataFrame.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.head())

print(df.head(3)) # To get multiple rows

```

1.3 take(n): Explicitly fetches n rows and returns them as a list of Row objects.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.take(2))

```

1.4 first(): Returns the first row of the DataFrame as a single Row object. It is similar to head(1) but directly gives one row instead of a list.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.first())

```

2. Inspecting Schema

Understanding the schema helps you see the structure of your data:

2.1 printSchema():Displays column names, data types, and nullability in a tree-like format.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.first())

```

2.2 dtypes: Returns a list of (column_name, data_type) pairs.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.dtypes)

```

2.3 schema: Returns the schema of the DataFrame as a StructType object. It includes column names, data types, and nullability information — useful when programmatically accessing fields.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.schema)

```

2.4. Columns: Returns a list of all column names in the DataFrame.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.columns)

```

2.5. Count(): Triggers a full scan of the DataFrame to return the total number of rows. It is an action and can be expensive on large data.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print(df.count())# counting the number of rows

```

3. Summary Statistics

3.1 df.describe(): Generates basic count, mean, stddev, min, max.It helps in quickly understanding data distribution. summary statistics for numerical columns such as:

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.describe().show()

```

3.2 summary(): Provides 25%, 50% (median), and 75%detailed descriptive statistics, along with percentiles like for DataFrame columns — including count, mean, stddev, min, max.

Advanced stats (Spark 3+):

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.summary().show()

```

4. Selecting Columns

4.1 Single column: To access a single column from a DataFrame, you can use:

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select("name").show()

```

4.2 Multiple columns: To select with multiple column names more than one column from a DataFrame, use select()

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select("name", "age").show()

```

4.3 Using expressions: You can apply column expressions inside select() to transform data — such as calculations, formatting, or renaming.

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select(col("age") + 5).show()

```

5. Filtering Data

Use filter() or where() to select rows that meet a condition (similar to SQL’s WHERE clause).

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.filter(df.age > 25).show()

df.where("age > 25").show()

```

6. Sorting Data

Ascending: Sort rows in ascending order from lowest to highest, use Order() or Sort() with the default order.

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.orderBy("age").show()

```

Descending: Sort data from highest to lowest, use OrderBy() or sort() with desc().

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.orderBy(col("age").desc()).show()

```

9. Removing Duplicates: Removes duplicate rows from the entire DataFrame.

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", 30),

("Rohan", 28),

("Alice", 25), # duplicate row

("Bob", 30)] # duplicate row

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.dropDuplicates().show()

```

Remove duplicates based on specific columns: Pass column names to dropDuplicates().

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", 30),

("Rohan", 28),

("Alice", 25), # duplicate row

("Bob", 30)] # duplicate row

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.dropDuplicates(["name"]).show()

```

7. Unique Values

Distinct(): Find distinct (unique) values in a specific column, use:

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", 30),

("Rohan", 28),

("Alice", 25), # duplicate row

("Bob", 30)] # duplicate row

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select("name").distinct().show()

```

Count unique values – Chain .count() after .distinct()

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", 30),

("Rohan", 28),

("Alice", 25), # duplicate row

("Bob", 30)] # duplicate row

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select("name").distinct().count()

```

8. Handling Null Values

Find nulls: Null values can cause errors or incorrect calculations.

The first step is to find where nulls exist.

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", None),

("Rohan", 28),

("Alice", 25),

("Bob", 30)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.filter(col("age").isNull()).show()

```

9. Checking Metadata

Number of partitions: Spark, data is divided into partitions To check how many partitions a DataFrame has for parallel processing we used rdd.getNumPartitions()

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", None),

("Rohan", 28),

("Alice", 25),

("Bob", 30)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print("Number of Partitions:", df.rdd.getNumPartitions())

```

len(df.columns): To find the number of columns in a DataFrame, count the column list

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("ExploreDF").getOrCreate()

data = [("Alice", 25),

("Bob", None),

("Rohan", 28),

("Alice", 25),

("Bob", 30)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

print("Number of Columns:", len(df.columns))

```

11. Understanding Execution Plan — explain()

Spark does lazy evaluation, meaning transformations don’t run immediately.
explain() helps you understand how Spark plans to execute your query before running it. It shows the logical plan and physical execution plan — including optimizations by the Catalyst optimizer.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExplainExample").getOrCreate()

data = [("Alice", 25),

("Bob", 30),

("Rohan", 22),

("Megha", 35)]

df = spark.createDataFrame(data, ["name", "age"])

df.filter(df.age > 25).explain(True)

```

Shows:

Logical plan
Optimized logical plan
Physical plan

This is important for performance tuning.

12. Converting to Pandas (small data only)

df.toPandas():Converts a Spark DataFrame into a Pandas DataFrame on the driver node. Never use this for big data — it loads everything into driver memory.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PandasConvert").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

pdf = df.toPandas()

print(pdf)

```

Exploring DataFrames

More Spark tutorials