DataFrame Creation in Pyspark
Spark tutorial · PySpark.in
PySpark DataFrame Creation — The Smart Way to Structure Big Data
Every PySpark program begins by creating a SparkSession, which is the entry point to Spark’s features—including DataFrames, Spark SQL, and streaming. Here’s how to initialize it:
```
from pyspark.sql import SparkSession
#create the spark session
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
print(spark)
```
What is a DataFrame in PySpark?
A DataFrame in PySpark is a distributed collection of data organized into named columns—conceptually similar to a table in a relational database or a DataFrame in R/Python. The key difference is that PySpark DataFrames are designed to handle massive data sets by distributing processing across multiple machines.
Key Features
- Distributed: DataFrames run across a cluster rather than a single machine.
- Schema-based: Each column has a name and a data type.
- Immutable: Transformations create new DataFrames instead of modifying existing ones.
- Lazy evaluated: Computation isn’t executed until you call an action (like show() or count()).
- Optimized: PySpark uses the Catalyst optimizer and Tungsten engine for efficient execution.
Ways to Create DataFrames in PySpark
Create DataFrame from Python List of Tuples
```
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
data = [("Alice", 25), ("Bob", 30), ("Charlie", 29)]
columns = ["name", "age"]
df = spark.createDataFrame(data, columns)
df.show()
```
Create DataFrame from Python Dictionaries
```
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
data = [{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]
df = spark.createDataFrame(data)
df.show()
```
Load DataFrame from CSV File
```
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
file_path = "s3a://pyspark420/uploads/2105dd49-0f23-4b20-9c64-93da8872aaff.csv"
df = spark.read.csv(file_path, header=True, inferSchema=True)
df.show()
```
Load DataFrame from Parquet
```
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
file_path = "s3a://pyspark420/uploads/sales/"
df = spark.read.parquet(file_path)
df.printSchema()
df.show()
```
Load DataFrame from JSON File
```
from pyspark.sql import SparkSession
# Create Spark session
spark = SparkSession.builder.appName("ReadMultiLineJSON").getOrCreate()
# File path (S3 or Local)
file_path = "s3a://pyspark420/uploads/products.json"
# Read JSON file
# multiline = true → tells Spark the JSON spans multiple lines (array or formatted JSON)
df = spark.read.option("multiline", "true").json(file_path)
# Show result without truncation
df.show(truncate=False)
```
Create Empty DataFrame with Schema
```
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
schema = StructType([
StructField("name", StringType(), True),
StructField("age", IntegerType(), True)
])
df = spark.createDataFrame([], schema)
df.printSchema()
```
From SQL / Database Table:
```
hide
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
df = spark.read.format("jdbc").options(
url="jdbc:postgresql://your-host:5432/yourdb",
driver="org.postgresql.Driver",
dbtable="public.employees",
user="username",
password="password"
).load()
df.show()
```
Read DataFrame from Hive Table
```
hide
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
# Make sure Hive support is enabled in your SparkSession
spark = SparkSession.builder \
.appName("HiveExample") \
.enableHiveSupport() \
.getOrCreate()
# Query a Hive table
df = spark.sql("SELECT * FROM default.employees")
df.show()
```
More Spark tutorials
- about pyspark
- text diagram
- Apache Spark Runtime Architecture
- Introduction to RDD
- Actions vs Transformations
- Lazy Evaluation in PySpark
All tutorials · Try the free PySpark compiler · Practice challenges