Rows & Columns in PySpark

Spark tutorial · PySpark.in

Rows & Columns in PySpark

Before filtering, joining, aggregating or transforming data, it’s essential to understand how PySpark represents individual records and fields. A DataFrame is a distributed table made up of Row objects, each with named fields. Unlike Pandas, where a column is simply a list of values, a PySpark Column is a logical expression used to reference or compute values across all rows

1. Working with Rows

A Row is a single record containing values for each column in the DataFrame. You can inspect or convert rows using a variety of methods:

• first() – Returns the first row as a Row object.
• head([n]) – Returns the first row or first n rows; if n is omitted, you get a single row.
• take(n)– Explicitly fetches the first n rows and returns them as a list of Row objects.
• asDict() – Converts a Row to a Python dictionary.
• Accessing fields – Use either index/key (row["name"]) or attribute (row.name) access.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("RowsColumnsTutorial").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

# Inspect individual rows

print(df.first()) # Single row

print(df.head()) # Same as first()

print(df.head(2)) # List of rows

print(df.take(3)) # List of rows (explicit)

row = df.first()

print(row.asDict()) # Convert to dict

print(row["name"], row.age) # Access fields

```

2. What is a Column in PySpark?

A Column is NOT a list of values like Pandas.

It is a logical expression that Spark applies on all rows.

Example of a column object:

```

hide

col("age")

```

Columns let Spark:

• Apply transformations
• Build execution plans
• Optimize queries

3. Selecting and Manipulating Columns

You can extract, rename, add, or remove columns using DataFrame methods:

• Selecting – Use select() with column names or expressions.
• Renaming – Use withColumnRenamed(old_name, new_name)..
• Adding new columns – Use withColumn(new_name, expression).
• Dropping columns – Use drop(column_name).

Example 1 — Select one column

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("RowsColumnsTutorial").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select("name").show() #select one column

df.select("name", "age").show()#select multiple column

df.select(col("age") + 5).show()#select with Expresion

```

Example 2 — Renaming Columns

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("RowsColumnsTutorial").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.withColumnRenamed("name", "full_name").show()

```

Example 5 — Adding New Columns

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("RowsColumnsTutorial").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.withColumn("age_plus_10", col("age") + 10).show()

```

Example 6 — Dropping Columns

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("RowsColumnsTutorial").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.drop("age").show()

```

Real-World Practical Examples: Combining column expressions with filters or actions:

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("RowsColumnsTutorial").getOrCreate()

data = [("Alice", 25), ("Bob", 30), ("Rohan", 28)]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.select(col("name"), col("age") + 1).show() #increase age of all rows

df.filter(col("age") > 25).show()#filter old people

df.collect() #convert entire data frame to python list

pdf = df.toPandas()#convert to Pandas

print(pdf)

```

4. Row vs Column (Simple Comparison Table)

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges