Pyspark Data Types

Spark tutorial · PySpark.in

Spark Data Types

Data types in PySpark determine how your data is stored, processed, and validated. Choosing the right types improves memory efficiency, execution speed, and the accuracy of your transformations and queries.

Why Data Types Matter

• Prevent data issues – A mismatched schema might read numbers as strings or dates as text, leading to errors.
• Optimize performance – Spark uses type information to create efficient query plans via its Catalyst optimize.
• Ensure correctness – You cannot apply numeric functions to string columns.
• Enable correct storage – Formats like Parquet, Delta, and ORC rely on proper type definitions.

2. Importing Spark Data Types

All Spark SQL data types live in the pyspark.sql.types module, and you can import them with a wildcard for convenience

```

hide

from pyspark.sql.types import *

```

This gives you access to primitive types (e.g., IntegerType, StringType), complex types (e.g., ArrayType, MapType, StructType), and auxiliary classes like StructField for schema definition.

3. Primitive Data Types

Primitive types represent simple scalar values. The table below summarizes the most common types and their Python equivalents:

Example: Creating a DataFrame with primitive types

```

from pyspark.sql import SparkSession

from pyspark.sql.types import *

spark = SparkSession.builder.appName("DataTypes").getOrCreate()

data = [("Alice", 25, True), ("Bob", 30, False)]

schema = StructType([

StructField("name", StringType(), True),

StructField("age", IntegerType(), True),

StructField("is_active", BooleanType(), True)

])

df = spark.createDataFrame(data, schema)

df.show()

df.printSchema()

```

4. Complex Data Types

Complex types allow you to represent nested or structured data, such as arrays, dictionaries, and nested objects.

4.1 ArrayType

Use ArrayType(elementType) when a column contains a list of values.

Example

```

from pyspark.sql.types import ArrayType, StringType

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataTypes").getOrCreate()

data = [("Alice", ["Python", "Spark"]),

("Bob", ["SQL", "Java"])]

schema = StructType([

StructField("name", StringType(), True),

StructField("skills", ArrayType(StringType()), True)

])

df = spark.createDataFrame(data, schema)

df.printSchema()

df.show(truncate=False)

```

4.2 MapType

MapType(keyType, valueType) is used for key–value pairs (similar to Python dictionaries).

Example

```

from pyspark.sql.types import MapType, StringType, IntegerType

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataTypes").getOrCreate()

data = [

("Alice", {"math": 85, "english": 95}),

("Bob", {"math": 75, "science": 80})

]

schema = StructType([

StructField("name", StringType(), True),

StructField("marks", MapType(StringType(), IntegerType()), True)

])

df = spark.createDataFrame(data, schema)

df.printSchema()

df.show(truncate=False)

```

4.3 StructType & StructField

StructType enables nested structures, frequently used when reading JSON files or defining nested columns manually.

Example

```

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataTypes").getOrCreate()

schema = StructType([

StructField("name", StringType(), True),

StructField("address", StructType([

StructField("city", StringType(), True),

StructField("pincode", IntegerType(), True)

]))

])

data = [

("Alice", ("Delhi", 110001)),

("Bob", ("Mumbai", 400001))

]

df = spark.createDataFrame(data, schema)

df.printSchema()

df.show(truncate=False)

```

Note on nullability: Each StructField has a nullable flag. Setting nullable=False enforces that column values cannot be null, aiding data quality.

5. Nullability in Spark Types

• Manual schemas: Defining a schema explicitly improves performance and accuracy. It avoids Spark’s automatic type inference, which can be slow or incorrect.
• Nullability: Each StructField has a nullable property. If nullable=False , Spark will not allow null values in that column.

Example

```

hide

StructField("name", StringType(), nullable=False)

```

Meaning:

• Spark will NOT allow NULL in this column.
• Useful for enforcing data quality.

6. Automatic Type Inference

If you don’t specify schema, Spark tries to guess and infer it:

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataTypes").getOrCreate()

df = spark.createDataFrame([("Alice", 25)])

df.printSchema()

```

However, type inference scans the entire dataset, which adds overhead; in production, prefer manual schemas for better performance

7. Casting (Changing Data Types)

Use .cast() on a column expression to convert its type—for instance, converting a string to an integer:

Example: Convert string to integer

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("CastingExample").getOrCreate()

data = [("Alice", "25"), ("Bob", "30"), ("Rohan", "28")]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.show()

# Convert string age → integer age

df2 = df.withColumn("age_int", col("age").cast("int"))

df2.show()

```

8. Checking Data Types

To inspect data types in your DataFrame:

• df.dtypes – Returns a list of (column_name, data_type) tuples.

• df.schema – Returns the full Spark StructType schema.

• df.printSchema() – Prints the schema in a tree format.

```

from pyspark.sql import SparkSession

from pyspark.sql.functions import col

spark = SparkSession.builder.appName("CastingExample").getOrCreate()

data = [("Alice", "25"), ("Bob", "30"), ("Rohan", "28")]

columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

df.show()

```

9. Real-World Example (Useful for ETL)

When reading a CSV, numeric values may be parsed as strings. You can cast them to the proper types for downstream processing:

```

from pyspark.sql.functions import col

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ETLExample").getOrCreate()

df = spark.read.csv("s3a://pyspark420/uploads/d11d21d3-fc70-4a99-9476-75f5e7ec19d2.csv", header=True)

df.printSchema()

df_clean = (

df

.withColumn("amount", col("amount").cast("double"))

.withColumn("quantity", col("quantity").cast("int"))

)

df_clean.printSchema()

df_clean.show()

```

Note:

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges