PySpark on AWS: S3, Glue and EMR with Examples

Spark tutorial · PySpark.in

To use PySpark on AWS, keep your data in Amazon S3 and choose where Spark runs: AWS Glue for serverless ETL jobs, Amazon EMR for clusters you control (or EMR Serverless), or your own machine with the S3A connector. The DataFrame code is the same everywhere; only the session setup and the path prefix change (s3:// on EMR and Glue, s3a:// elsewhere). This tutorial shows each step with code.

ServiceWhat it isUse it when
Amazon S3Object storage for your data lake (CSV, JSON, Parquet, Delta)Always: it is where the data lives
AWS GlueServerless Spark jobs plus the Glue Data CatalogScheduled ETL without managing servers; catalog-driven pipelines
Amazon EMRManaged Spark/Hadoop clusters on EC2Large or long-running workloads, custom Spark settings, notebooks on a cluster
EMR ServerlessSpark jobs without a cluster to sizeEMR features with pay-per-job billing
Local / Docker + S3AOpen-source Spark reading S3 over the s3a:// connectorDevelopment and tests against real S3 data

New to Spark itself? Start with how Spark runs your code and installing PySpark locally, then come back here.

Read S3 data in PySpark (s3a://)

Outside AWS, Spark reads S3 through the Hadoop S3A connector. Add the hadoop-aws package whose version matches the Hadoop bundled with your Spark (a mismatch is the most common cause of ClassNotFoundException or NoSuchMethodError), and let the default credentials chain find your keys:

```

hide

import os

from pyspark.sql import SparkSession

 

# Outside AWS (laptop, Docker, Kubernetes): add the S3A connector that matches Spark's Hadoop.

# Spark 3.5 ships Hadoop 3.3.4, so use hadoop-aws 3.3.4. On EMR and Glue skip this: use s3:// paths.

spark = (SparkSession.builder

.appName("ReadFromS3")

.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4")

.config("spark.hadoop.fs.s3a.aws.credentials.provider",

"com.amazonaws.auth.DefaultAWSCredentialsProviderChain")

.getOrCreate())

# Credentials come from the provider chain: AWS_PROFILE / AWS_ACCESS_KEY_ID in the environment,

# ~/.aws/credentials, or the IAM role of the machine. Never hard-code keys in the script.

```

On Amazon EMR and AWS Glue you do not add anything: S3 support (EMRFS) is built in and the cluster or job uses its IAM role. Use s3:// paths there. Now read files as if they were local:

```

hide

# CSV: header row, schema inferred (inferSchema reads the data twice: give a schema for big files)

csv_df = (spark.read

.option("header", True)

.option("inferSchema", True)

.csv("s3a://my-bucket/raw/orders/*.csv"))

 

json_df = spark.read.json("s3a://my-bucket/raw/events/") # every JSON file in the prefix

parquet_df = spark.read.parquet("s3a://my-bucket/curated/sales/") # schema stored in the files

 

csv_df.printSchema()

csv_df.show(5)

```

Spark splits the files and reads them in parallel, one task per file split. Read a whole prefix or use wildcards (s3a://my-bucket/raw/orders/*.csv) instead of looping over files in Python. If the data is partitioned into folders such as year=2024/month=06/, filtering on those columns makes Spark read only the matching folders.

Write PySpark DataFrames to S3

Writing uses the same df.write API with an S3 path. Partition large tables by the columns you filter on, and keep Snappy-compressed Parquet as the default format for analytics:

```

hide

# mode: "overwrite", "append", "ignore" or "error" (the default)

df.write.mode("overwrite").option("header", True).csv("s3a://my-bucket/out/orders_csv/")

df.write.mode("overwrite").json("s3a://my-bucket/out/orders_json/")

df.write.mode("overwrite").parquet("s3a://my-bucket/out/orders_parquet/")

 

# Partitioned, compressed Parquet: one folder per year/month value

(df.write.mode("overwrite")

.option("compression", "snappy")

.partitionBy("year", "month")

.parquet("s3a://my-bucket/curated/sales/"))

```

Try it: see the partition layout Spark writes

This runs here without an AWS account. It writes the same partitioned Parquet layout you would get in a bucket, then reads back one month; in the plan, PartitionFilters shows that Spark only opens the year=2024/month=6 folder.

```python

import os, tempfile

from pyspark.sql import SparkSession

 

spark = SparkSession.builder.getOrCreate()

sales = spark.createDataFrame(

[(2024, 6, "keyboard", 40.0), (2024, 6, "mouse", 15.0),

(2024, 7, "monitor", 180.0), (2025, 1, "laptop", 900.0)],

["year", "month", "item", "amount"])

 

# The same call you would make with an s3a:// or s3:// path; here a local folder

out = os.path.join(tempfile.mkdtemp(), "sales")

sales.write.mode("overwrite").partitionBy("year", "month").parquet(out)

 

# One folder per partition value: the layout you get in an S3 bucket

for root, dirs, files in sorted(os.walk(out)):

parquet_files = [f for f in files if f.endswith(".parquet")]

if parquet_files:

print(os.path.relpath(root, out), "->", len(parquet_files), "file(s)")

 

# Filtering on partition columns reads only the matching folder (see PartitionFilters)

june = spark.read.parquet(out).filter("year = 2024 AND month = 6")

june.show()

june.explain()

```

Each partition folder holds one or more Parquet files. Thousands of tiny files slow down both S3 and Spark, so control the file count before writing with df.repartition(n, "year", "month") or df.coalesce(n). On EMR 5.20.0 and later, Parquet writes use the EMRFS S3-optimized committer by default, which avoids slow rename steps.

AWS Glue PySpark example: GlueContext and DynamicFrame

AWS Glue runs your PySpark script on managed, serverless Spark. Every Glue job starts the same way: read the job arguments, wrap the SparkContext in a GlueContext, and initialise the Job object (it powers job bookmarks):

```

hide

import sys

from awsglue.utils import getResolvedOptions

from awsglue.context import GlueContext

from awsglue.job import Job

from pyspark.context import SparkContext

 

args = getResolvedOptions(sys.argv, ["JOB_NAME"])

sc = SparkContext()

glueContext = GlueContext(sc) # Glue-aware wrapper around Spark

spark = glueContext.spark_session # a normal SparkSession: spark.read, spark.sql all work

job = Job(glueContext)

job.init(args["JOB_NAME"], args) # needed for job bookmarks

```

A typical extract, transform and load job reads a table from the Glue Data Catalog, cleans it with the DataFrame API and writes partitioned Parquet back to S3:

```

hide

from awsglue.dynamicframe import DynamicFrame

 

# Extract: a table registered in the Glue Data Catalog (by a crawler or a previous job)

dyf = glueContext.create_dynamic_frame.from_catalog(

database="sales_db",

table_name="orders_raw",

transformation_ctx="orders_raw", # lets job bookmarks skip files already processed

)

 

# Transform with the DataFrame API, then convert back

df = dyf.toDF()

clean = df.filter(df["amount"] > 0).dropDuplicates(["order_id"])

dyf_clean = DynamicFrame.fromDF(clean, glueContext, "orders_clean")

 

# Load: partitioned Parquet in S3

glueContext.write_dynamic_frame.from_options(

frame=dyf_clean,

connection_type="s3",

format="parquet",

connection_options={"path": "s3://my-bucket/curated/orders/", "partitionKeys": ["order_date"]},

format_options={"compression": "snappy"},

)

job.commit()

```

You can also write a DynamicFrame straight to S3 with a Glue sink; glueparquet is Glue's optimised Parquet writer:

```

hide

# Inside a Glue job: write a DynamicFrame to S3 as partitioned Parquet

glueContext.write_dynamic_frame.from_options(

frame=dynamic_frame,

connection_type="s3",

format="glueparquet",

connection_options={"path": "s3://my-bucket/curated/events/", "partitionKeys": ["event_type"]},

format_options={"compression": "snappy"},

)

```

DynamicFrame vs DataFrame

A DynamicFrame is Glue's own record set: each record describes itself, so it copes with messy data where a column has different types in different files, and it works with Glue features such as job bookmarks and catalog writes. A Spark DataFrame has one fixed schema and the full Spark SQL API. In practice you read and write with DynamicFrames and do the heavy transformation as a DataFrame, converting with toDF() and DynamicFrame.fromDF().

To create the job, open AWS Glue, create a Spark job with Python, paste the script, and give the job an IAM role that can read and write your S3 paths and use the Data Catalog. Turn on job bookmarks to process only new files on each run.

Use the Glue Data Catalog with Spark SQL

The Glue Data Catalog can act as Spark's Hive metastore, so tables created by crawlers, Athena or other jobs are visible to Spark SQL. In a Glue job, pass the job parameter --enable-glue-datacatalog. On EMR, add this configuration when you create the cluster (or tick "Use AWS Glue Data Catalog for Spark table metadata"):

```

hide

[

{

"Classification": "spark-hive-site",

"Properties": {

"hive.metastore.client.factory.class":

"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"

}

}

]

```

With the catalog enabled, query its tables with Spark SQL:

```

hide

spark.sql("USE sales_db")

spark.sql("SHOW TABLES").show()

spark.sql("SELECT * FROM orders_raw LIMIT 5").show()

```

Inside a Glue job you can also turn a catalog table into a temporary view and use SQL on it:

```

hide

dyf = glueContext.create_dynamic_frame.from_catalog(database="sales_db", table_name="orders_raw")

dyf.toDF().createOrReplaceTempView("orders")

spark.sql("SELECT status, count(*) AS orders FROM orders GROUP BY status").show()

```

Both Spark SQL and Glue ETL scripts then work on the same tables. Tables in unusual formats may need the matching SerDe on the classpath.

Run PySpark on Amazon EMR

Amazon EMR runs Spark on a cluster of EC2 instances that it creates, configures and can resize for you:

  1. Create a cluster with an EMR 6.x or 7.x release and Spark selected. Pick instance types and counts, an EC2 key pair if you want SSH, and an S3 location for logs. Enable the Glue Data Catalog option if you use it.
  2. Upload your script to S3, for example s3://my-bucket/scripts/sales_job.py.
  3. Submit a step: in the console open the cluster, choose Steps, then Add step, type "Spark application", the S3 path of the script and its arguments. EMR runs spark-submit for you; cluster deploy mode is the usual choice.
  4. Check the result: step logs go to your S3 log location, and the Spark UI (from the EMR console) shows jobs, stages and executors.

The same step from the AWS CLI:

```

hide

aws emr add-steps --cluster-id j-XXXXXXXXXXXXX \

--steps 'Type=Spark,Name=Daily sales job,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,s3://my-bucket/scripts/sales_job.py,--run-date,2026-11-19]'

```

Or, on the primary node, call spark-submit directly:

```

hide

spark-submit \

--deploy-mode cluster \

--master yarn \

s3://my-bucket/scripts/sales_job.py \

--run-date 2026-11-19

```

EMR 6.x and 7.x use Python 3 by default. Install extra Python packages with a bootstrap action so every node has them.

S3 performance tips for PySpark

FAQ: PySpark with AWS

How do I read a file from S3 in PySpark? On EMR or Glue, call spark.read.parquet("s3://bucket/path/") (or .csv, .json). Elsewhere, add the hadoop-aws package that matches your Spark's Hadoop version and use s3a:// paths; credentials come from your AWS profile, environment or IAM role.

What is the difference between s3:// and s3a://? On EMR and Glue, s3:// uses EMRFS, Amazon's built-in connector, and is the recommended scheme there. Open-source Spark outside AWS uses the Hadoop S3A connector with s3a://.

Should I use AWS Glue or EMR for PySpark? Glue for scheduled ETL when you do not want to manage servers; EMR when you need control over Spark versions and settings, very large or long-running jobs, or a shared cluster for notebooks. EMR Serverless sits in between.

Is AWS Glue just PySpark? Glue ETL jobs run Apache Spark, and you write them in PySpark (or Scala). Glue adds DynamicFrames, the Data Catalog, crawlers and job bookmarks on top.

Can I practise PySpark for AWS without an AWS account? Yes. The DataFrame code is the same; only the paths change. Run the partitioning example above here, or use the online PySpark compiler, then point the same code at S3.

Keep learning

More Spark tutorials

PySpark tutorial: all lessons · All tutorials · Try the free PySpark compiler · Practice challenges