PySpark: read a CSV into a DataFrame

In PySpark you load a CSV with spark.read.csv(path) or spark.read.option(...).csv(path). Set header=True to use the first row as column names and inferSchema=True to detect types automatically — or pass an explicit schema for speed and correctness in production.

Header and schema

spark.read.option("header", True).option("inferSchema", True).csv("data.csv") reads names and infers types. inferSchema scans the file, so for large data provide an explicit StructType schema instead.

Common options

Useful options: sep (delimiter), nullValue, quote, escape, mode (PERMISSIVE / DROPMALFORMED / FAILFAST) to control how malformed rows are handled.

Schema inference versus an explicit schema

inferSchema=True makes Spark read the file an extra time to work out column types, which doubles the I/O and can still guess wrong — a ZIP code becomes an integer and loses its leading zero, or a mixed column collapses to string. For anything you run more than once, define a StructType and pass schema=my_schema. It is faster because the file is read once, and it is safer because the types are pinned. Set header=True so the first row becomes column names rather than data.

Malformed rows, multi-line fields and the rescue column

Real CSVs are messy. The mode option decides what happens to rows that do not parse: PERMISSIVE (the default) keeps them and nulls the bad fields, DROPMALFORMED discards them, and FAILFAST aborts the read. Silent data loss usually comes from leaving the default in place and never checking. If quoted values contain line breaks, set multiLine=True or rows will split in the wrong place. Use columnNameOfCorruptRecord (or the rescued data column) to capture what failed so you can inspect it instead of losing it, and set the correct escape and sep characters rather than assuming commas and double quotes.

Run without a file, then load your own CSV

The worked example below parses two in-memory CSV records using from_csv(), so it needs no upload or filesystem access. For a real file, open Data in the compiler, upload a small CSV and use its generated read code. A placeholder such as /path/to/data.csv does not refer to a file on the server. File-reader header and multiline options do not apply to this single-record example.

CSV options: file reader and record parser
Need	File reader	from_csv record parser
Column names	header=True or explicit schema	Explicit schema
Data types	Explicit schema preferred; inferSchema available	Explicit schema
Multiline records	multiLine=True where needed	One complete record per input value
Separator	sep option	sep option

Example (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_csv, col

spark = SparkSession.builder.getOrCreate()
records = spark.createDataFrame([("1,Alice,120",), ("2,Bob,90",)], ["csv_record"])
schema = "id INT, name STRING, amount INT"
result = records.select(from_csv(col("csv_record"), schema).alias("row")).select("row.*")
for row in result.orderBy("id").collect():
    print(row.id, row.name, row.amount)

Parses two CSV records with an explicit schema and prints deterministic output. This demonstrates CSV parsing; it does not read a file. Use the Data upload workflow for file-reader examples.

Run this example in the free online PySpark compiler

Expected output

1 Alice 120
2 Bob 90

Frequently asked questions

How do I read a CSV file in PySpark?

Use spark.read.option("header", True).csv("path.csv"). Add .option("inferSchema", True) to auto-detect column types.

What does inferSchema do and is it slow?

inferSchema scans the data to detect column types. It's convenient but adds an extra pass over the file; for large or production data, supply an explicit schema instead.

How do I read a CSV with a different delimiter?

Set the sep option, e.g. spark.read.option("sep", ";").csv(path) for a semicolon-delimited file.

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs