Hadoop to PySpark

Spark tutorial · PySpark.in

Why PySpark?

In the early days, tools like Hadoop MapReduce were widely used to process large datasets. However, they were slow because data had to be repeatedly read from and written to disk at every stage.

PySpark (Apache Spark + Python) changed this. By storing most data in memory (RAM) instead of disk, PySpark can run computations 10–100x faster.

What Is PySpark Used For?

PySpark is a distributed data processing framework. It allows you to:

Analyze very large datasets quickly that don't fit in single memory.
Use Python-like syntax, making it easier for developers.
Distribute work across multiple computers (nodes) instead of overloading a single machine.

Think of it like teamwork:

Normal Python: One person does the entire job.
PySpark: A team works together, each handling part of the task, then combining results.

Example: Average Age Calculation

Imagine you have a huge file containing client information and you want the average age:

Without PySpark (Python only): The file might not even fit into RAM, causing your computer to crash.
With PySpark: The file is split across multiple machines, each machine processes its chunk, and the results are combined efficiently.

Scenario	Without PySpark	With PySpark
Input File	May not fit in memory → crash	Split across multiple machines
Processing	Slow, sequential	Fast, parallel
Result	May fail	Result computed efficiently

Why Spark Came After Hadoop?

Originally, businesses used Hadoop + MapReduce to process large data. It worked, but had limitations:

MapReduce always read/write from disk → slow.
Batch-only → couldn’t handle real-time data.
Tightly coupled with Hadoop → limited flexibility.

Apache Spark was created as a faster, flexible alternative to solve these issues.

Components of Hadoop (for context)

HDFS (Hadoop Distributed File System): Stores massive files across low-cost machines.
YARN (Yet Another Resource Negotiator): Manages cluster resources and runs jobs (MapReduce, Spark, etc.).

Feature	Spark	MapReduce (Hadoop)
Speed	10–100x faster	Slower (disk-based)
Mode	Batch + Real-time	Batch only
Works With	S3, HDFS, JDBC, Kafka, Cassandra, etc.	HDFS only
Cluster Managers	YARN, Kubernetes, Standalone	YARN only

Types of Processing in Spark

Batch Processing: Handle data in chunks (e.g., daily reports).
Stream Processing: Process data in real time (e.g., live tweets).
Data Ingestion: Import data from sources like HDFS, AWS, Kafka, or Twitter.

Spark Ecosystem (Key Components)

Spark Core: Handles unstructured data and basic processing.
Spark SQL: Works with structured/semi-structured data (tables, CSVs, JSON).
Spark Streaming: Real-time data processing.
Spark MLlib: Machine learning algorithms at scale.
Spark GraphX: Graph processing (e.g., social networks).

Note: Spark Core and Spark SQL are most common for batch workloads.

$C:\Users\legion\AppData\Local\Microsoft\Windows\INetCache\Content.MSO\BC6FACA.tmp$ Spark Integration

Spark integrates seamlessly with modern ecosystems:

Cloud: AWS, Azure, Google Cloud.
Databases & Storage: Cassandra, MongoDB, HBase, HDFS, S3.
Streaming Systems: Kafka, Flume, Kinesis.

Compared to MapReduce, which works only with HDFS + YARN, Spark is far more versatile.

$C:\Users\legion\AppData\Local\Microsoft\Windows\INetCache\Content.MSO\4AC58D68.tmp$

Hadoop to PySpark

More Spark tutorials