Spark Revolution

Spark tutorial · PySpark.in

Understanding Data Lake Landscape

Before HDFS & MapReduce, we had Data warehouses

(like Teradata, Exadata) where the data is brought from many OLTP/OLAP systems.

The challenges faced by data warehouses are as follows -

Vertical Scaling - adding more DW was expensive

Large Capital Investment

Storage - non-scalable

Support only structured data

To overcome above challenges, Data Lake came into the picture with following features -

Horizontal Scaling - adding more cheap servers to clusters

Low Capital Investment

Storage - scalable (cloud storage)

Support structured, unstructured and semi-structured data

Data Lake Architecture

The core capability of data lake was storage but with timely advancement, it had developed 4 important capabilities –

Data collection and ingestion (Ingest)
Data storage and management (Storage)
Data processing and transformation (Process)
Data access and retrieval (Consume)

Let's analyse the below Data Lake architecture for understanding different layers.

Apache Spark &it's Eco-system:

Apache Spark is a unified analytics engine for large-scale distributed data processing and machine learning.
It is an open-source cluster computing framework which handles both batch data & streaming data.
Spark was built on the top of the Hadoop MapReduce.
Spark provides in-memory storage for intermediate computations whereas alternative approaches like Hadoop's MapReduce writes data to and from computer hard drives. So, Spark process the data much faster than other alternatives.

History of Spark :

The Spark was initiated by Matei Zaharia at UC Berkeley's AMP Lab in 2009. It was open sourced in 2010 under a BSD license.

In 2013, the project was acquired by Apache Software Foundation. In 2014, the Spark emerged as a Top-Level Apache Project.

Spark eco-system and it's components :

The Apache Spark ecosystem may be divided into three tiers as indicated in the above diagram.

Storage and Cluster Manager
Spark Core
Set of Libraries

Storage and Cluster Manager :

Apache Spark is a distributed processing engine. However, it doesn't come with an inbuilt cluster resource manager and a distributedstorage system.
There is a good reason behind that design decision. Apache Spark tried to decouple the functionality of a cluster resource manager,distributed storage and a distributed computing engine from the beginning.
This design allows us to use Apache Spark with any compatible cluster manager and storage solution. Hence, the storage and thecluster manager are part of the ecosystem however they are not part of Apache Spark.
You can plugin a cluster manager and a storage system of your choice. There are multiple alternatives. You can use Apache YARN,Mesos, and even Kubernetes as a cluster manager for Apache Spark. Similarly, for the storage system, you can use HDFS, Amazon S3,Azure Data Lake, Google Cloud storage, Cassandra File system and many others.

2) Spark Core :

Apache Spark core contains two main components -

1) Spark Compute engine

2) Spark Core APIs

Spark Compute engine -

The Spark Core includes a computation engine for Spark. Basic functions including memory management, job scheduling, fault recovery, and most crucially, communication with the cluster manager and storage system, are provided by the compute engine.
So, in order to give the user a smooth experience, the Spark compute engine manages and executes our Spark jobs. Simply submit your jobto Spark, and the core of Spark does the rest.

Spark Core APIs –

Structured API
Unstructured API

The Structured APIs consists of data frames and data sets. They are designed and optimized to work with structured data.

The Unstructured APIs are the lower level APIs including RDDs, Accumulators and Broadcast variables. These core APIs are available in Scala, Python, Java, and R.

3) Set of Libraries :

Apache Spark has different set of libraries and packages that make it a powerfull big data processing framework. The set of libraries includeSpark SQL, Spark Streaming, Spark MLlib, and Spark GraphX.

These libraries provide different functionalities for data processing, analysis, and machine learning tasks.

Working :

Spark SQL - Allows you to use SQL queries for structured data processing.

Spark Streaming - Helps you to consume and process continuous data streams.

MLlib - A machine learning library that delivers high-quality algorithms.

GraphX - Comes with a library of typical graph algorithms.

These libraries offer us APIs, DSLs, and algorithms in multiple languages. They directly depend on Spark Core APIs to achieve distributed processing.

Spark Revolution

Understanding Data Lake Landscape

More Spark tutorials