Data Engineering Foundations and Core Concepts

Spark tutorial · PySpark.in

1. What is Data Engineering?

Data Engineering is the practice of designing, building, testing and maintaining the infrastructure and systems that collect, store, process and make data available for analytics, machine learning and business intelligence.

2. Goals of Data Engineering:

3. Key Components of Data Engineering:

Component

Purpose

Data Ingestion

Collect data from various sources (Batch / Streaming)

Data Storage

Store raw and processed data (Data Lake / Warehouse / DB)

Data Processing

Clean, transform and enrich data

Orchestration

Schedule, automate and monitor data workflows

Data Quality

Validate, test and ensure accuracy and consistency

Serving Layer

Provide data for analytics, ML, dashboards, reports

Governance & Security

Manage access, lineage, catalog, compliance and security

4. Types of Data:

By Structure:

By Processing:

5. Common Data Sources:

6. Data Engineering Workflow (High Level):

Source Systems → Ingestion (Extract) → Storage (Data Lake) → Processing (Transform) → Serving (Analytics / ML) → Monitoring & Optimization

7. Data Storage Options Ans:

Storage Type

Examples

Use Cases

Pros

Cons

Relational DB

MySQL, PostgreSQL, SQL Server

OLTP, Transactions, Operational Data

ACID compliant, Structured, Reliable

Not good for big data, Less flexible

Data Warehouse

Snowflake, BigQuery, Redshift, Synapse

Analytics, Reporting, BI

Optimized for ML, Scalable, SQL support

Expensive for raw data, Requires modeling

Data Lake

AWS S3, ADLS, GCS, HDFS

Raw data storage, ML, Exploration

Low cost, Scalable, Stores any format

No schema enforcement, Needs processing

NoSQL DB

MongoDB, Cassandra, DynamoDB

High volume, Unstructured, Real-time apps

Flexible schema, High availability

Limited joins, Eventual consistency

Object Storage

Amazon S3, Azure Blob

Backup, Archive, Data Lake

Durable, Low cost, Highly scalable

Not for direct querying

8. Data Formats Ans:

9. Batch vs Streaming Ans:

Batch Processing

Streaming Processing

Process data in chunks at scheduled intervals

Process data in real-time as it is generated

Higher latency (minutes to hours)

Low latency (seconds to milliseconds)

Simpler to implement

Requires state management

Use cases: Reports, ETL jobs, Historical analysis

Use cases: Fraud detection, Real-time dashboards

Tools: Airflow, Spark (Batch)

Tools: Kafka, Spark Streaming, Flink, Kinesis

10. ETL vs ELT Ans:

ETL (Extract, Transform, Load)

ELT (Extract, Load, Transform)

Data is extracted from source

Data is extracted and loaded first

Transformed in staging area

Transformations done in target system

Cleaned and aggregated

Leverages power of target system

Loaded to Data Warehouse

Good for big data and cloud platforms

Good for small to medium datasets

Modern approach

Traditional approach


11. OLTP vs OLAP

Ans:

OLTP (Online Transaction Processing)

OLAP (Online Analytical Processing)

Handles daily transactions

Handles analytical queries

High volume of small transactions

Complex queries over large data

Fast processing, simple queries

Aggregated, summarized data

Normalized data

Denormalized (Star / Snowflake schema)

Example: Banking, E-commerce

Example: Reporting, Dashboards, BI

DB: MySQL, PostgreSQL, Oracle

DB: Snowflake, Redshift, BigQuery

12. Data Modeling (Concepts) Ans:

  • ER Model: Entities, Attributes, Relationships
  • Normalization: 1NF, 2NF, 3NF – reduces redundancy
  • Denormalization: Improves query performance in analytics
  • Schema Types: Star Schema, Snowflake Schema, Fact & Dimension Tables

Star Schema (Example)

13. Data Quality & Best Practices Ans:

  • Accuracy – Data is correct
  • Completeness – No missing data
  • Consistency – Data is uniform across sources
  • Timeliness – Data is up to date
  • Validity – Data follows rules & formats
  • Uniqueness – No duplicate records
  • Profiling – Understand data patterns
  • Validation Rules – Range, Null checks, Regex
  • Deduplication – Remove duplicates
  • Monitoring – Alerts and metrics
  • Data Governance – Ownership, lineage, policy

14. Data Partitioning & Indexing Ans:

Partitioning (Improve performance):

  • Horizontal partitioning – Split rows
  • Vertical partitioning – Split columns
  • Types: Range, Hash, List, Composite
  • Helps in faster query & manage large data

Indexing (Speed up queries):

  • Index is a data structure
  • Types: Primary, Secondary, Clustered, Non-clustered, Composite Index
  • Improves read performance, slows down writes

15. Common ETL Pipeline Stages Ans:

Extract → Validate → Cleanse → Transform → Load → Monitor

  • Extract: Pull data from various sources
  • Validate: Check data quality & completeness
  • Cleanse: Handle nulls, duplicates, incorrect data
  • Transform: Apply business rules, aggregations, enrich data
  • Load: Load to target (DB, DW, Data Lake)
  • Monitor: Monitor pipeline, data quality, alerts

16. Apache Spark Fundamentals Ans:

  • Unified analytics engine for big data
  • In-memory computation – Faster processing
  • Supports: Batch, Streaming, ML, Graph
  • Core Components:
    • Spark Core – RDD, DAG, Scheduler
    • Spark SQL – Structured data processing
    • Spark Streaming – Real-time data
    • MLlib – Machine learning
    • GraphX – Graph processing
  • Languages: Scala, Python (PySpark), Java R, SQL
  • Key Concepts:
    • RDD – Resilient Distributed Dataset
    • DataFrame – Distributed collection
    • Lazy Evaluation
    • Actions vs Transformations
    • Caching & Persistence

More Spark tutorials

All tutorials · Try the free PySpark compiler · Practice challenges