Lakehouse Architecture

Spark tutorial · PySpark.in

Lakehouse Architecture with Delta Lake

Lakehouse architecture unifies the scalability of data lakes with the performance and reliability of data warehouses, providing a single platform for all data and analytics. Traditional data warehouses store cleaned, structured data in predefined schemas optimized for fast BI queries, but they are expensive and rigid. Data lakes, by contrast, allow storing vast amounts of raw, unstructured or semi-structured data at low cost, but they lack built‑in governance and fast query features. The lakehouse model was introduced to combine these strengths: it keeps raw data in cheap object storage yet adds warehouse-like features (ACID transactions, schemas, indexing) for reliability and performance.

Figure: The data warehouse, data lake, and modern data lakehouse architectures

In a lakehouse, data of all types (structured tables, logs, images, JSON, etc.) can coexist. Users can query raw data directly with SQL or machine learning frameworks without separate ETL pipelines. Lakehouses support schema enforcement, versioning, and governance so that data doesn’t become a “data swamp.” In practice, a lakehouse supports ACID transactions (for reliable updates and deletes), time travel (querying past versions of data), and unified batch/streaming processing. As one tutorial explains, “Lakehouse platforms combine the scale and flexibility of a data lake with the reliability and performance of a data warehouse”. By integrating these capabilities, lakehouses let organizations scale massively while still powering high-concurrency analytics and even real-time workloads.

What is Delta Lake?

Delta Lake is an open-source storage layer that implements the lakehouse architecture on Apache Spark and other engines. Delta Lake sits on top of cloud object stores (like AWS S3, Azure Blob) and Parquet files, adding a transactional log and advanced features. Essentially, a Delta table is just Parquet data plus a special “_delta_log” folder that records all changes and metadata. Delta Lake “extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling”. It’s fully compatible with Spark APIs (and APIs for Python, Java, etc.) and was designed so you can use the same data for both batch and streaming queries. In short, Delta Lake brings the reliability of databases and warehouses into a flexible data lake.

Delta Lake was created to solve the problems of raw data lakes. Traditional lakes have no built‑in transactions or schema enforcement, so concurrent writes can corrupt data and outdated or mismatched files can slip in. Delta Lake addresses these by adding ACID transactions, schema enforcement/evolution, versioning (time travel), and upsert/delete support. For example, Delta Lake ensures any write to a table is atomic and isolated, preventing partial or conflicting writes. It also records every transaction (insert, update, delete) in its log, so you can audit changes or even “travel back” to a prior state of the table. As one summary puts it: “Delta Lake enhances traditional Data Lakes by adding ACID transactions, schema enforcement, versioning, and time-travel capabilities, addressing challenges like data consistency, governance, and performance.”. This makes lakes built on Delta suitable for mission-critical analytics and high-concurrency workloads.

Delta Lake Architecture and Features

A Delta table’s storage structure is illustrated below:

Figure: A Delta Lake table consists of Parquet data files and a transaction log in the _delta_log folder.

As shown above, each Delta table has one or more Parquet data files (storing the actual rows) and a _delta_log directory of JSON/Parquet logs. Every time data is written or changed, Delta Lake writes a new log entry. These log files form a transaction history. This transaction log is the heart of Delta Lake: it enables ACID semantics, time travel, and efficient incremental processing.

Key Delta Lake features include:

ACID Transactions: Every write to a Delta table (whether batch or streaming) is atomic and isolated. This means concurrent jobs can append, merge, or update data safely without conflicts. Spark queries see a consistent snapshot of the table. Delta Lake’s serializable isolation guarantees the strongest data consistency, akin to a database.
Schema Enforcement and Evolution: Delta Lake enforces schemas on write. By default, any new file added to a table must match the table’s schema, preventing “bad data” (e.g. missing or mismatched columns) from corrupting the table. You can also evolve the schema (adding new columns) in a controlled way. This ensures data quality while still being flexible.
Time Travel (Versioning): Because every change is logged, Delta Lake can query historical versions of data. You can easily “roll back” to an earlier snapshot or audit what the data looked like at a given date/time or version number. This is useful for debugging, auditing, or reproducing experiments. For example, reading a table with option("versionAsOf", oldVersion) lets you see its prior contents.
Unified Batch + Streaming: Delta Lake natively supports exactly-once streaming writes. You can write streaming data into a Delta table just like batch data, and any new data appears in real time to queries. Likewise, you can read a Delta table as a streaming source. This unifies ETL pipelines: the same table can be appended incrementally or updated in micro-batches, eliminating the need for separate “stream” and “batch” storage layers.
Scalable Metadata: Unlike plain Parquet files, Delta Lake uses an index of file metadata. This allows very large tables (petabytes, billions of files) to be managed efficiently. Spark can quickly list only relevant files for a query (using partition filters, data skipping) rather than scanning everything.
DML Support (DELETE, UPDATE, MERGE): Delta Lake provides easy APIs to delete rows, update values, and upsert (merge) datasets. For example, you can do deltaTable.delete(condition = expr(...)) or use the powerful MERGE statement to handle complex upserts. These operations modify the table in-place without needing to rewrite the entire dataset manually (unlike a naïve Parquet approach). All changes are recorded in the transaction log.

Overall, Delta Lake implements a “lakehouse” storage format. It preserves the open, low-cost storage of data lakes, while adding warehouse-like transactional guarantees and performance features. Many organizations use Delta Lake as the foundation of their lakehouse because of this rich feature set.

Lakehouse Architecture