Install PySpark on Windows

Spark tutorial · PySpark.in

Install PySpark on Windows

Step 1: Install Visual Studio Code (VS Code)

Download and install VS Code from Visual Studio Code.

After installation:

• Open VS Code
• Go to Extensions (Ctrl + Shift + X)
• Install the Python extension by Microsoft

Step 2: Install Java (JDK 8)

Download OpenJDK 8 from Adoptium Temurin 8.

Install it. Example installation path:

C:\Program Files\Java\jdk1.8.0_361

Set environment variables

1. Press Win + R
2. Type sysdm.cpl
3. Open Advanced → Environment Variables

Under System variables, create:

JAVA_HOME = C:\Program Files\Java\jdk1.8.0_361

Edit the Path variable and add:

%JAVA_HOME%\bin

Verify Java installation

Open Command Prompt and run:

java -version

Step 3: Install Python

Download Python 3.10 from Python Downloads.

During installation, make sure to check:

Add Python to PATH

Verify Python installation

python --version

=

Step 4: Install PySpark

Spark / PySpark and Java compatibility

For this tutorial, we’ll use PySpark 3.5.

Install PySpark:

pip install pyspark==3.5.0

Step 5: Install Winutils (Windows only)

For some Windows setups, PySpark may require winutils.exe.

Download from winutils GitHub.

Extract it to a folder such as:

C:\hadoop\hadoop-3.0.0

Make sure this file exists:

C:\hadoop\hadoop-3.0.0\bin\winutils.exe

Set environment variables

Create:

HADOOP_HOME = C:\hadoop\hadoop-3.0.0

Add to Path:

%HADOOP_HOME%\bin

Step 6: Test PySpark

Create a file named pyspark_test.py in VS Code and add:

```

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TestPySpark").getOrCreate()

data = [("Alice", 29), ("Bob", 35)]

df = spark.createDataFrame(data, ["Name", "Age"])

df.show()

spark.stop()

```

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges