PySpark on AWS: S3, Glue and EMR with Examples
Spark tutorial · PySpark.in
To use PySpark on AWS, keep your data in Amazon S3 and choose where Spark runs: AWS Glue for serverless ETL jobs, Amazon EMR for clusters you control (or EMR Serverless), or your own machine with the S3A connector. The DataFrame code is the same everywhere; only the session setup and the path prefix change (s3:// on EMR and Glue, s3a:// elsewhere). This tutorial shows each step with code.
| Service | What it is | Use it when |
|---|---|---|
| Amazon S3 | Object storage for your data lake (CSV, JSON, Parquet, Delta) | Always: it is where the data lives |
| AWS Glue | Serverless Spark jobs plus the Glue Data Catalog | Scheduled ETL without managing servers; catalog-driven pipelines |
| Amazon EMR | Managed Spark/Hadoop clusters on EC2 | Large or long-running workloads, custom Spark settings, notebooks on a cluster |
| EMR Serverless | Spark jobs without a cluster to size | EMR features with pay-per-job billing |
| Local / Docker + S3A | Open-source Spark reading S3 over the s3a:// connector | Development and tests against real S3 data |
New to Spark itself? Start with how Spark runs your code and installing PySpark locally, then come back here.
Read S3 data in PySpark (s3a://)
Outside AWS, Spark reads S3 through the Hadoop S3A connector. Add the hadoop-aws package whose version matches the Hadoop bundled with your Spark (a mismatch is the most common cause of ClassNotFoundException or NoSuchMethodError), and let the default credentials chain find your keys:
```
hide
import os
from pyspark.sql import SparkSession
# Outside AWS (laptop, Docker, Kubernetes): add the S3A connector that matches Spark's Hadoop.
# Spark 3.5 ships Hadoop 3.3.4, so use hadoop-aws 3.3.4. On EMR and Glue skip this: use s3:// paths.
spark = (SparkSession.builder
.appName("ReadFromS3")
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4")
.config("spark.hadoop.fs.s3a.aws.credentials.provider",
"com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
.getOrCreate())
# Credentials come from the provider chain: AWS_PROFILE / AWS_ACCESS_KEY_ID in the environment,
# ~/.aws/credentials, or the IAM role of the machine. Never hard-code keys in the script.
```
On Amazon EMR and AWS Glue you do not add anything: S3 support (EMRFS) is built in and the cluster or job uses its IAM role. Use s3:// paths there. Now read files as if they were local:
```
hide
# CSV: header row, schema inferred (inferSchema reads the data twice: give a schema for big files)
csv_df = (spark.read
.option("header", True)
.option("inferSchema", True)
.csv("s3a://my-bucket/raw/orders/*.csv"))
json_df = spark.read.json("s3a://my-bucket/raw/events/") # every JSON file in the prefix
parquet_df = spark.read.parquet("s3a://my-bucket/curated/sales/") # schema stored in the files
csv_df.printSchema()
csv_df.show(5)
```
Spark splits the files and reads them in parallel, one task per file split. Read a whole prefix or use wildcards (s3a://my-bucket/raw/orders/*.csv) instead of looping over files in Python. If the data is partitioned into folders such as year=2024/month=06/, filtering on those columns makes Spark read only the matching folders.
Write PySpark DataFrames to S3
Writing uses the same df.write API with an S3 path. Partition large tables by the columns you filter on, and keep Snappy-compressed Parquet as the default format for analytics:
```
hide
# mode: "overwrite", "append", "ignore" or "error" (the default)
df.write.mode("overwrite").option("header", True).csv("s3a://my-bucket/out/orders_csv/")
df.write.mode("overwrite").json("s3a://my-bucket/out/orders_json/")
df.write.mode("overwrite").parquet("s3a://my-bucket/out/orders_parquet/")
# Partitioned, compressed Parquet: one folder per year/month value
(df.write.mode("overwrite")
.option("compression", "snappy")
.partitionBy("year", "month")
.parquet("s3a://my-bucket/curated/sales/"))
```
Try it: see the partition layout Spark writes
This runs here without an AWS account. It writes the same partitioned Parquet layout you would get in a bucket, then reads back one month; in the plan, PartitionFilters shows that Spark only opens the year=2024/month=6 folder.
```python
import os, tempfile
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
sales = spark.createDataFrame(
[(2024, 6, "keyboard", 40.0), (2024, 6, "mouse", 15.0),
(2024, 7, "monitor", 180.0), (2025, 1, "laptop", 900.0)],
["year", "month", "item", "amount"])
# The same call you would make with an s3a:// or s3:// path; here a local folder
out = os.path.join(tempfile.mkdtemp(), "sales")
sales.write.mode("overwrite").partitionBy("year", "month").parquet(out)
# One folder per partition value: the layout you get in an S3 bucket
for root, dirs, files in sorted(os.walk(out)):
parquet_files = [f for f in files if f.endswith(".parquet")]
if parquet_files:
print(os.path.relpath(root, out), "->", len(parquet_files), "file(s)")
# Filtering on partition columns reads only the matching folder (see PartitionFilters)
june = spark.read.parquet(out).filter("year = 2024 AND month = 6")
june.show()
june.explain()
```
Each partition folder holds one or more Parquet files. Thousands of tiny files slow down both S3 and Spark, so control the file count before writing with df.repartition(n, "year", "month") or df.coalesce(n). On EMR 5.20.0 and later, Parquet writes use the EMRFS S3-optimized committer by default, which avoids slow rename steps.
AWS Glue PySpark example: GlueContext and DynamicFrame
AWS Glue runs your PySpark script on managed, serverless Spark. Every Glue job starts the same way: read the job arguments, wrap the SparkContext in a GlueContext, and initialise the Job object (it powers job bookmarks):
```
hide
import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc) # Glue-aware wrapper around Spark
spark = glueContext.spark_session # a normal SparkSession: spark.read, spark.sql all work
job = Job(glueContext)
job.init(args["JOB_NAME"], args) # needed for job bookmarks
```
A typical extract, transform and load job reads a table from the Glue Data Catalog, cleans it with the DataFrame API and writes partitioned Parquet back to S3:
```
hide
from awsglue.dynamicframe import DynamicFrame
# Extract: a table registered in the Glue Data Catalog (by a crawler or a previous job)
dyf = glueContext.create_dynamic_frame.from_catalog(
database="sales_db",
table_name="orders_raw",
transformation_ctx="orders_raw", # lets job bookmarks skip files already processed
)
# Transform with the DataFrame API, then convert back
df = dyf.toDF()
clean = df.filter(df["amount"] > 0).dropDuplicates(["order_id"])
dyf_clean = DynamicFrame.fromDF(clean, glueContext, "orders_clean")
# Load: partitioned Parquet in S3
glueContext.write_dynamic_frame.from_options(
frame=dyf_clean,
connection_type="s3",
format="parquet",
connection_options={"path": "s3://my-bucket/curated/orders/", "partitionKeys": ["order_date"]},
format_options={"compression": "snappy"},
)
job.commit()
```
You can also write a DynamicFrame straight to S3 with a Glue sink; glueparquet is Glue's optimised Parquet writer:
```
hide
# Inside a Glue job: write a DynamicFrame to S3 as partitioned Parquet
glueContext.write_dynamic_frame.from_options(
frame=dynamic_frame,
connection_type="s3",
format="glueparquet",
connection_options={"path": "s3://my-bucket/curated/events/", "partitionKeys": ["event_type"]},
format_options={"compression": "snappy"},
)
```
DynamicFrame vs DataFrame
A DynamicFrame is Glue's own record set: each record describes itself, so it copes with messy data where a column has different types in different files, and it works with Glue features such as job bookmarks and catalog writes. A Spark DataFrame has one fixed schema and the full Spark SQL API. In practice you read and write with DynamicFrames and do the heavy transformation as a DataFrame, converting with toDF() and DynamicFrame.fromDF().
To create the job, open AWS Glue, create a Spark job with Python, paste the script, and give the job an IAM role that can read and write your S3 paths and use the Data Catalog. Turn on job bookmarks to process only new files on each run.
Use the Glue Data Catalog with Spark SQL
The Glue Data Catalog can act as Spark's Hive metastore, so tables created by crawlers, Athena or other jobs are visible to Spark SQL. In a Glue job, pass the job parameter --enable-glue-datacatalog. On EMR, add this configuration when you create the cluster (or tick "Use AWS Glue Data Catalog for Spark table metadata"):
```
hide
[
{
"Classification": "spark-hive-site",
"Properties": {
"hive.metastore.client.factory.class":
"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
}
}
]
```
With the catalog enabled, query its tables with Spark SQL:
```
hide
spark.sql("USE sales_db")
spark.sql("SHOW TABLES").show()
spark.sql("SELECT * FROM orders_raw LIMIT 5").show()
```
Inside a Glue job you can also turn a catalog table into a temporary view and use SQL on it:
```
hide
dyf = glueContext.create_dynamic_frame.from_catalog(database="sales_db", table_name="orders_raw")
dyf.toDF().createOrReplaceTempView("orders")
spark.sql("SELECT status, count(*) AS orders FROM orders GROUP BY status").show()
```
Both Spark SQL and Glue ETL scripts then work on the same tables. Tables in unusual formats may need the matching SerDe on the classpath.
Run PySpark on Amazon EMR
Amazon EMR runs Spark on a cluster of EC2 instances that it creates, configures and can resize for you:
- Create a cluster with an EMR 6.x or 7.x release and Spark selected. Pick instance types and counts, an EC2 key pair if you want SSH, and an S3 location for logs. Enable the Glue Data Catalog option if you use it.
- Upload your script to S3, for example
s3://my-bucket/scripts/sales_job.py. - Submit a step: in the console open the cluster, choose Steps, then Add step, type "Spark application", the S3 path of the script and its arguments. EMR runs
spark-submitfor you; cluster deploy mode is the usual choice. - Check the result: step logs go to your S3 log location, and the Spark UI (from the EMR console) shows jobs, stages and executors.
The same step from the AWS CLI:
```
hide
aws emr add-steps --cluster-id j-XXXXXXXXXXXXX \
--steps 'Type=Spark,Name=Daily sales job,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,s3://my-bucket/scripts/sales_job.py,--run-date,2026-11-19]'
```
Or, on the primary node, call spark-submit directly:
```
hide
spark-submit \
--deploy-mode cluster \
--master yarn \
s3://my-bucket/scripts/sales_job.py \
--run-date 2026-11-19
```
EMR 6.x and 7.x use Python 3 by default. Install extra Python packages with a bootstrap action so every node has them.
S3 performance tips for PySpark
- Avoid many small files. Aim for files of roughly 128 MB to 1 GB. Compact small files with a periodic job (read and rewrite with
repartition) ors3-dist-cpon EMR. - Partition by what you filter on, such as date. Filters on partition columns let Spark skip whole folders.
- Control the number of output files with
repartitionorcoalescebefore writing; every file is at least one S3 request when writing and reading. - Use an S3-aware committer. EMR uses the EMRFS S3-optimized committer for Parquet by default. Outside EMR, use the S3A committers (
fs.s3a.committer.nameset todirectoryormagic, with Spark's hadoop-cloud module). Do not rely onFileOutputCommitteralgorithm version 2: it still renames objects and is not safe on S3. - Spread load across prefixes. S3 supports at least 3,500 write and 5,500 read requests per second per prefix, so partitioned folders also spread the request load.
- Throttle very wide listings. Discovering thousands of partition folders runs many parallel list calls; lower
spark.sql.sources.parallelPartitionDiscovery.parallelismif you see S3 503 SlowDown errors. - Let retries work. EMRFS and S3A retry throttled requests with backoff; on EMR you can raise
fs.s3.maxRetriesfor very busy buckets. - Use Glue job bookmarks (and bounded execution) so each run reads only new data.
FAQ: PySpark with AWS
How do I read a file from S3 in PySpark? On EMR or Glue, call spark.read.parquet("s3://bucket/path/") (or .csv, .json). Elsewhere, add the hadoop-aws package that matches your Spark's Hadoop version and use s3a:// paths; credentials come from your AWS profile, environment or IAM role.
What is the difference between s3:// and s3a://? On EMR and Glue, s3:// uses EMRFS, Amazon's built-in connector, and is the recommended scheme there. Open-source Spark outside AWS uses the Hadoop S3A connector with s3a://.
Should I use AWS Glue or EMR for PySpark? Glue for scheduled ETL when you do not want to manage servers; EMR when you need control over Spark versions and settings, very large or long-running jobs, or a shared cluster for notebooks. EMR Serverless sits in between.
Is AWS Glue just PySpark? Glue ETL jobs run Apache Spark, and you write them in PySpark (or Scala). Glue adds DynamicFrames, the Data Catalog, crawlers and job bookmarks on top.
Can I practise PySpark for AWS without an AWS account? Yes. The DataFrame code is the same; only the paths change. Run the partitioning example above here, or use the online PySpark compiler, then point the same code at S3.
Keep learning
- Apache Spark architecture: driver, executors and DAG
- Spark SQL tutorial: run SQL queries in PySpark
- PySpark practice challenges: graded problems you can run in the browser.
- AWS and Databricks certification guides, including AWS Certified Developer and Solutions Architect.
More Spark tutorials
- DataFrame Operations in PySpark
- Spark SQL Tutorial: Run SQL Queries in PySpark
- Spark SQL Queries in PySpark: spark.sql() Examples
- Registering UDFs in SQL
- PySpark Joins
- Broadcast Join and Skew Join in PySpark
PySpark tutorial: all lessons · All tutorials · Try the free PySpark compiler · Practice challenges