PySpark: read a CSV into a DataFrame

In PySpark you load a CSV with `spark.read.csv(path)` or `spark.read.option(...).csv(path)`. Set `header=True` to use the first row as column names and `inferSchema=True` to detect types automatically — or pass an explicit schema for speed and correctness in production.

Header and schema

spark.read.option("header", True).option("inferSchema", True).csv("data.csv") reads names and infers types. inferSchema scans the file, so for large data provide an explicit StructType schema instead.

Common options

Useful options: sep (delimiter), nullValue, quote, escape, mode (PERMISSIVE / DROPMALFORMED / FAILFAST) to control how malformed rows are handled.

Schema inference versus an explicit schema

inferSchema=True makes Spark read the file an extra time to work out column types, which doubles the I/O and can still guess wrong — a ZIP code becomes an integer and loses its leading zero, or a mixed column collapses to string. For anything you run more than once, define a StructType and pass schema=my_schema. It is faster because the file is read once, and it is safer because the types are pinned. Set header=True so the first row becomes column names rather than data.

Malformed rows, multi-line fields and the rescue column

Real CSVs are messy. The mode option decides what happens to rows that do not parse: PERMISSIVE (the default) keeps them and nulls the bad fields, DROPMALFORMED discards them, and FAILFAST aborts the read. Silent data loss usually comes from leaving the default in place and never checking. If quoted values contain line breaks, set multiLine=True or rows will split in the wrong place. Use columnNameOfCorruptRecord (or the rescued data column) to capture what failed so you can inspect it instead of losing it, and set the correct escape and sep characters rather than assuming commas and double quotes.

Example (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = (
    spark.read
         .option("header", True)
         .option("inferSchema", True)
         .csv("path/to/data.csv")
)
df.printSchema()
df.show(5)

Reads a CSV using the header row for names and inferring column types.

Run this example in the free online PySpark compiler

Frequently asked questions

How do I read a CSV file in PySpark?

Use spark.read.option("header", True).csv("path.csv"). Add .option("inferSchema", True) to auto-detect column types.

What does inferSchema do and is it slow?

inferSchema scans the data to detect column types. It's convenient but adds an extra pass over the file; for large or production data, supply an explicit schema instead.

How do I read a CSV with a different delimiter?

Set the sep option, e.g. spark.read.option("sep", ";").csv(path) for a semicolon-delimited file.

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs