Install PySpark on Windows 10/11 with VS Code

Spark tutorial · PySpark.in

To install PySpark on Windows you need three things: Java (JDK 17), Python (3.10 or 3.11) and the pyspark package from pip. This step-by-step guide sets them up for Windows 10 and 11, runs PySpark in VS Code, and fixes the errors Windows users hit most often.

No time to install?: You can run PySpark in your browser with no setup in the free online PySpark compiler. Install locally when you want to work on your own files and projects.

What you need

ComponentRecommended versionWhy
Java (JDK)Temurin JDK 17Spark runs on the JVM. Java 17 works with PySpark 3.5 and is required by Spark 4.
Python3.10 or 3.11Well supported by PySpark 3.5 and by most data libraries.
PySpark3.5.x from pipIncludes Spark itself; no separate Spark download is needed.
winutils.exe + hadoop.dllBuilt for Hadoop 3.3Only needed to write files on Windows; PySpark 3.5 bundles Hadoop 3.3.
VS CodeLatest, with the Python and Jupyter extensionsEditor, terminal and notebooks in one place.

Step 1: Install Java (JDK 17)

  1. Download the Temurin JDK 17 Windows installer (.msi) from the Adoptium website.
  2. Run it. On the Custom Setup screen, enable Set JAVA_HOME variable and Add to PATH; that saves you from setting them by hand.
  3. Open a new Command Prompt and check the version:

```

hide

java -version

echo %JAVA_HOME%

```

You should see openjdk version "17..." and a folder such as C:\Program Files\Eclipse Adoptium\jdk-17.... If JAVA_HOME is empty, set it yourself: press Win + R, type sysdm.cpl, open Advanced > Environment Variables, add a system variable JAVA_HOME pointing to the JDK folder (not its bin folder), and add %JAVA_HOME%\bin to Path.

Avoid: paths with spaces and brackets in custom locations, and having several JDKs on Path; Spark uses the first java it finds. If java -version shows an old version, move the JDK 17 entry to the top of Path.

Step 2: Install Python

  1. Download Python 3.11 (or 3.10) for Windows from python.org.
  2. In the installer, tick Add python.exe to PATH, then choose Install Now.
  3. Check it in a new Command Prompt:

```

hide

python --version

pip --version

```

Microsoft Store alias: If python opens the Microsoft Store, turn off the App execution aliases for python.exe and python3.exe in Windows Settings, or use the py launcher instead.

Step 3: Install PySpark with pip

Use a virtual environment so PySpark and its dependencies stay separate from other projects. In a folder for your project, run:

```

hide

python -m venv .venv

.venv\Scripts\activate

python -m pip install --upgrade pip

pip install "pyspark==3.5.*"

```

The quotes matter in Command Prompt and PowerShell. pyspark==3.5.* installs the newest 3.5 patch release. Check it:

```

hide

python -c "import pyspark; print(pyspark.__version__)"

```

If PowerShell refuses to run Activate.ps1, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once, or use Command Prompt.

Step 4: Add winutils.exe and hadoop.dll (for writing files)

Reading data and running DataFrame code usually works without this step. Writing files on Windows (for example df.write.parquet(...)) needs Hadoop's Windows helpers:

  1. Download winutils.exe and hadoop.dll built for Hadoop 3.3 from a winutils repository on GitHub.
  2. Put both files in C:\hadoop\bin.
  3. Set the system variable HADOOP_HOME to C:\hadoop and add %HADOOP_HOME%\bin to Path.
  4. Open a new terminal (or restart VS Code) so the new variables are picked up.

Step 5: Set up VS Code for PySpark

  1. Install VS Code, then the Python and Jupyter extensions from Microsoft (Ctrl + Shift + X).
  2. Open your project folder with File > Open Folder.
  3. Press Ctrl + Shift + P, run Python: Select Interpreter, and pick the interpreter inside .venv.
  4. Open a terminal in VS Code (Ctrl + `); it should show (.venv) at the start of the prompt.

Point Spark at the same Python as VS Code, so the driver and the workers match. Add these two user environment variables, or put them at the top of your script before creating the session:

```

hide

import os, sys

os.environ["PYSPARK_PYTHON"] = sys.executable

os.environ["PYSPARK_DRIVER_PYTHON"] = sys.executable

```

Step 6: Test your installation

Create pyspark_test.py in VS Code with the code below and run it with the Run button or python pyspark_test.py. You can also press Run on this page to see the expected output.

```python

from pyspark.sql import SparkSession, functions as F

 

spark = SparkSession.builder.appName("TestPySpark").master("local[*]").getOrCreate()

print("Spark version:", spark.version)

 

data = [("Alice", 29, "Pune"), ("Bob", 35, "Delhi"), ("Asha", 41, "Pune")]

df = spark.createDataFrame(data, ["name", "age", "city"])

df.show()

 

df.groupBy("city").agg(F.avg("age").alias("avg_age")).orderBy("city").show()

 

spark.stop()

```

The first run takes a few seconds while the JVM starts. If you see the two tables, PySpark is installed correctly. The Spark web UI is at http://localhost:4040 while the session is running.

To use a notebook instead, create test.ipynb in VS Code, select the .venv kernel, and run the same code in a cell (VS Code may ask to install ipykernel; accept).

Common PySpark errors on Windows and how to fix them

"Java gateway process exited before sending its port number"

Spark could not start Java. Check that java -version works in the same terminal, that JAVA_HOME points to the JDK folder (not bin), and restart VS Code after changing environment variables.

"HADOOP_HOME and hadoop.home.dir are unset" or "Could not locate executable null\bin\winutils.exe"

Complete Step 4: winutils.exe in C:\hadoop\bin and HADOOP_HOME=C:\hadoop. This warning is harmless for reading data but writing files fails without it.

"UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0" when writing

hadoop.dll is missing or built for the wrong Hadoop version. Put the Hadoop 3.3 hadoop.dll next to winutils.exe in C:\hadoop\bin and make sure that folder is on Path.

"Python worker failed to connect back" or "Python was not found"

Spark started a different Python (often the Microsoft Store alias). Set PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to your interpreter as in Step 5, and turn off the python.exe app execution alias.

"java.lang.UnsupportedClassVersionError" or other Java version errors

The Java version does not match the Spark version. Use JDK 17 for PySpark 3.5 and Spark 4, and make sure an older Java is not earlier on Path.

"pip is not recognized"

Python was installed without the PATH option. Use python -m pip install ..., or rerun the installer and tick Add python.exe to PATH.

Which Java version does each Spark version need?

Spark / PySparkSupported Java
3.3.xJava 8, 11, 17
3.4.xJava 8, 11, 17
3.5.xJava 8 (8u371 or later), 11, 17
4.0.xJava 17, 21

When in doubt, use JDK 17: it works for every current PySpark 3.5 and 4.x release.

Next steps

More Spark tutorials

PySpark tutorial: all lessons · All tutorials · Try the free PySpark compiler · Practice challenges