Install PySpark on Windows 10/11 with VS Code
Spark tutorial · PySpark.in
To install PySpark on Windows you need three things: Java (JDK 17), Python (3.10 or 3.11) and the pyspark package from pip. This step-by-step guide sets them up for Windows 10 and 11, runs PySpark in VS Code, and fixes the errors Windows users hit most often.
No time to install?: You can run PySpark in your browser with no setup in the free online PySpark compiler. Install locally when you want to work on your own files and projects.
What you need
| Component | Recommended version | Why |
|---|---|---|
| Java (JDK) | Temurin JDK 17 | Spark runs on the JVM. Java 17 works with PySpark 3.5 and is required by Spark 4. |
| Python | 3.10 or 3.11 | Well supported by PySpark 3.5 and by most data libraries. |
| PySpark | 3.5.x from pip | Includes Spark itself; no separate Spark download is needed. |
| winutils.exe + hadoop.dll | Built for Hadoop 3.3 | Only needed to write files on Windows; PySpark 3.5 bundles Hadoop 3.3. |
| VS Code | Latest, with the Python and Jupyter extensions | Editor, terminal and notebooks in one place. |
Step 1: Install Java (JDK 17)
- Download the Temurin JDK 17 Windows installer (.msi) from the Adoptium website.
- Run it. On the Custom Setup screen, enable Set JAVA_HOME variable and Add to PATH; that saves you from setting them by hand.
- Open a new Command Prompt and check the version:
```
hide
java -version
echo %JAVA_HOME%
```
You should see openjdk version "17..." and a folder such as C:\Program Files\Eclipse Adoptium\jdk-17.... If JAVA_HOME is empty, set it yourself: press Win + R, type sysdm.cpl, open Advanced > Environment Variables, add a system variable JAVA_HOME pointing to the JDK folder (not its bin folder), and add %JAVA_HOME%\bin to Path.
Avoid: paths with spaces and brackets in custom locations, and having several JDKs on Path; Spark uses the first java it finds. If java -version shows an old version, move the JDK 17 entry to the top of Path.
Step 2: Install Python
- Download Python 3.11 (or 3.10) for Windows from python.org.
- In the installer, tick Add python.exe to PATH, then choose Install Now.
- Check it in a new Command Prompt:
```
hide
python --version
pip --version
```
Microsoft Store alias: If python opens the Microsoft Store, turn off the App execution aliases for python.exe and python3.exe in Windows Settings, or use the py launcher instead.
Step 3: Install PySpark with pip
Use a virtual environment so PySpark and its dependencies stay separate from other projects. In a folder for your project, run:
```
hide
python -m venv .venv
.venv\Scripts\activate
python -m pip install --upgrade pip
pip install "pyspark==3.5.*"
```
The quotes matter in Command Prompt and PowerShell. pyspark==3.5.* installs the newest 3.5 patch release. Check it:
```
hide
python -c "import pyspark; print(pyspark.__version__)"
```
If PowerShell refuses to run Activate.ps1, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once, or use Command Prompt.
Step 4: Add winutils.exe and hadoop.dll (for writing files)
Reading data and running DataFrame code usually works without this step. Writing files on Windows (for example df.write.parquet(...)) needs Hadoop's Windows helpers:
- Download
winutils.exeandhadoop.dllbuilt for Hadoop 3.3 from a winutils repository on GitHub. - Put both files in
C:\hadoop\bin. - Set the system variable
HADOOP_HOMEtoC:\hadoopand add%HADOOP_HOME%\bintoPath. - Open a new terminal (or restart VS Code) so the new variables are picked up.
Step 5: Set up VS Code for PySpark
- Install VS Code, then the Python and Jupyter extensions from Microsoft (Ctrl + Shift + X).
- Open your project folder with File > Open Folder.
- Press Ctrl + Shift + P, run Python: Select Interpreter, and pick the interpreter inside
.venv. - Open a terminal in VS Code (Ctrl + `); it should show
(.venv)at the start of the prompt.
Point Spark at the same Python as VS Code, so the driver and the workers match. Add these two user environment variables, or put them at the top of your script before creating the session:
```
hide
import os, sys
os.environ["PYSPARK_PYTHON"] = sys.executable
os.environ["PYSPARK_DRIVER_PYTHON"] = sys.executable
```
Step 6: Test your installation
Create pyspark_test.py in VS Code with the code below and run it with the Run button or python pyspark_test.py. You can also press Run on this page to see the expected output.
```python
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("TestPySpark").master("local[*]").getOrCreate()
print("Spark version:", spark.version)
data = [("Alice", 29, "Pune"), ("Bob", 35, "Delhi"), ("Asha", 41, "Pune")]
df = spark.createDataFrame(data, ["name", "age", "city"])
df.show()
df.groupBy("city").agg(F.avg("age").alias("avg_age")).orderBy("city").show()
spark.stop()
```
The first run takes a few seconds while the JVM starts. If you see the two tables, PySpark is installed correctly. The Spark web UI is at http://localhost:4040 while the session is running.
To use a notebook instead, create test.ipynb in VS Code, select the .venv kernel, and run the same code in a cell (VS Code may ask to install ipykernel; accept).
Common PySpark errors on Windows and how to fix them
"Java gateway process exited before sending its port number"
Spark could not start Java. Check that java -version works in the same terminal, that JAVA_HOME points to the JDK folder (not bin), and restart VS Code after changing environment variables.
"HADOOP_HOME and hadoop.home.dir are unset" or "Could not locate executable null\bin\winutils.exe"
Complete Step 4: winutils.exe in C:\hadoop\bin and HADOOP_HOME=C:\hadoop. This warning is harmless for reading data but writing files fails without it.
"UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0" when writing
hadoop.dll is missing or built for the wrong Hadoop version. Put the Hadoop 3.3 hadoop.dll next to winutils.exe in C:\hadoop\bin and make sure that folder is on Path.
"Python worker failed to connect back" or "Python was not found"
Spark started a different Python (often the Microsoft Store alias). Set PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to your interpreter as in Step 5, and turn off the python.exe app execution alias.
"java.lang.UnsupportedClassVersionError" or other Java version errors
The Java version does not match the Spark version. Use JDK 17 for PySpark 3.5 and Spark 4, and make sure an older Java is not earlier on Path.
"pip is not recognized"
Python was installed without the PATH option. Use python -m pip install ..., or rerun the installer and tick Add python.exe to PATH.
Which Java version does each Spark version need?
| Spark / PySpark | Supported Java |
|---|---|
| 3.3.x | Java 8, 11, 17 |
| 3.4.x | Java 8, 11, 17 |
| 3.5.x | Java 8 (8u371 or later), 11, 17 |
| 4.0.x | Java 17, 21 |
When in doubt, use JDK 17: it works for every current PySpark 3.5 and 4.x release.
Next steps
- Apache Spark architecture: what runs where when you call
getOrCreate(). - Read a CSV file in PySpark: load your first real dataset.
- PySpark practice challenges: practise DataFrame skills with instant feedback.
- Online PySpark compiler: try code without any local setup.
More Spark tutorials
- DataFrame Operations in PySpark
- Spark SQL Tutorial: Run SQL Queries in PySpark
- Spark SQL Queries in PySpark: spark.sql() Examples
- Registering UDFs in SQL
- PySpark Joins
- Broadcast Join and Skew Join in PySpark
PySpark tutorial: all lessons · All tutorials · Try the free PySpark compiler · Practice challenges