The Genesis of Spark

Spark tutorial · PySpark.in

The Genesis of Spark

Increased consumer traffic, a variety of new forms of data and greater computations demanded the need for more storage and better performance. Traditional data storage methods including relational database management systems (RDBMSs) and imperative programming techniques were unable to handle the enormous amounts of data and their processing.

Google is the first to overcome below problems-

Data collection and ingestion
Data storage and management
Data processing and transformation
Data access and retrieval

Google published the white papers in a sequence to solve these issues –

The Google white papers were highly appreciated by the opensource community and served as the inspiration for the design and development of a comparable open source implementation, called

Hadoop.

Hadoop is an open-source software framework for storing and processing large amounts of data in a distributed computing environment.
It is designed to handle big data and is based on the MapReduce programming model, which allows for the parallel processing of large datasets.
Its framework is based on Java programming.
It facilitates to start with the small clusters and expand the size as you grow.
It allows the storage capacity of 100's to 1000's of computers and use it as unified storage system

Hive:

Many solutions have been developed over Hadoop platform by various organizations.
Some of the widely adopted systems were Hive, Pig & HBase.
Apache Hive is the most popular adopted component of Hadoop.

Hive offered following core capabilities on Hadoop platform –

Create

Databases
Tables
Views

Run SQL Queries

Bringing together, Hadoop as platform and Hive as a database became very popular. But we still had other problems –

Performance - Hive SQL query performing slower than RDBMS SQL query

Ease of Development - writing MapReduce program was difficult

Language Support - MapReduce was only available in JAVA

Storage - expensive than cloud storage

Resource Management - only YARN container support, unable to use other container like Mesos, Docker , Kubernetes , etc

The point is, Hadoop left a lot scope for improvement and as a result Apache Spark came into the existence...!

The Genesis of Spark

The Genesis of Spark

Hive:

More Spark tutorials