Pitfalls of Data Lakes

Spark tutorial · PySpark.in

Understanding Delta Lake: From Data Warehouses to the Lakehouse

Data architecture has evolved from rigid data warehouses to flexible data lakes, and now to unified data lakehouses. Delta Lake is an open-source storage layer that brings reliability and performance back to data lakes, enabling a true lakehouse architecture. In this tutorial we’ll explain what Delta Lake is and why it was invented, stepping through traditional data warehouse designs, the rise of data lakes (and their limitations), and how Delta Lake unifies the best of both worlds. We use clear diagrams and examples, and provide technical depth for intermediate learners.

Traditional Data Warehouses: Architecture, Pros, and Cons

In a traditional data warehouse, an organization pre-defines a strict schema and ingests cleaned, structured data optimized for analytics. These systems typically use a three-tier architecture:

• Bottom tier: The data warehouse database server itself, storing cleansed data from various sources.
• Middle tier: An OLAP (Online Analytical Processing) server or engine that transforms data and builds multidimensional schemas for fast queries.
• Top tier: BI/reporting tools where analysts query and visualize the data.

Figure: A classic three-tier data warehouse architecture (Source: Panoply).

This architecture enforces schema-on-write: every record must conform to the predefined schema before loading. This ensures high data quality and consistent, fast query performance. For example, enterprise warehouses can handle thousands of SQL queries per day with sub-second response times on well-modeled data. The downside is inflexibility: changing requirements or ingesting unstructured data (logs, JSON, images, etc.) is hard, since every new field requires altering the schema. Traditional warehouses are also expensive to scale on commodity hardware, and cannot natively ingest real-time or semi-structured streams.

Key points: Data warehouses deliver ACID transactions, fast queries, and strong governance on structured data. However, they impose upfront schemas (less flexible) and often rely on batch ETL to load data.

Data Lakes: Flexible Storage, New Challenges

In contrast, a data lake is a large-scale storage repository that holds raw data in its native formats (CSV, JSON, images, logs, etc.) until needed. Data lakes allow schema-on-read: you can ingest data first and apply structure only when querying. This flexibility makes them ideal for exploratory analytics and machine learning on semi-structured or unstructured data. For instance, Internet of Things (IoT) streams, sensor logs, and text files can all flow into the lake without transformation.

Figure: A data lake stores vast volumes of raw data in object storage and feeds analytics, ML, and BI tools.

Data lakes have two main advantages: low cost (storage on cheap object stores like S3) and high flexibility (can hold any data type). But they come with serious challenges. A raw data lake lacks ACID transactions and built‑in data management. There’s no native support for updates, deletes or merges, and schema enforcement is weak. This often leads to "data swamp" scenarios: missing or corrupted records go undetected, duplicate files accumulate, and it’s hard to know which files are valid. For example, with a simple cloud data lake, “deleted” files often linger for days, causing stale data to be read unless manually vacuumed.

Common pitfalls of raw data lakes include (but are not limited to): no ACID guarantees, no built-in DML (UPDATE/DELETE/MERGE), easy corruption on failed writes, schema mismatches, and no versioning of historical data. As one Delta Lake tutorial notes, data engineers may need to reprocess entire data pipelines whenever a write job fails, since intermediate partial writes are not rolled back. Ensuring data quality and regulatory compliance (e.g. GDPR deletion) is especially painful on a plain data lake. Because you cannot atomically delete individual user records in a swarm of Parquet files, fulfilling a data removal request can require expensive, error-prone manual ETL steps.

Note: offer scalability and flexibility (schema-on-read, any data), but lack transactional guarantees and governance. Traditional data lakes do not support reliable deletes or updates, making them unreliable for production analytics.

The Lakehouse Architecture: Unifying Warehouses and Lakes

To bridge these worlds, the data lakehouse architecture has emerged. A lakehouse combines the best of data warehouses (governance, ACID, performance) with the flexibility of data lakes. In a lakehouse, the raw storage is typically cloud object storage, but an additional transaction layer (like Delta Lake) on top provides database-like features. Queries (via SQL or BI tools) can run directly on the same storage that powers ML and streaming pipelines, removing the silo between warehouses and lakes.

Figure: A modern lakehouse architecture contains layers for ingestion, storage, metadata, APIs, and consumption. Storage uses a format like Delta Lake to unify batch and streaming data.

In a lakehouse, data engineers can implement medallion layers (bronze/silver/gold) on top of raw data, apply schema enforcement, and still serve BI dashboards from the same tables used for ML. Databricks coins the term “lakehouse” to describe this pattern: offering reliability and performance of data warehouses together with the openness and flexibility of data lakes. With the lakehouse, you no longer need separate ETL copies for warehouses and lakes – the single copy of data can feed both traditional analytics and AI.

Note: The lakehouse is an open architecture using tools like Delta Lake to give data lakes ACID properties and fast queries, effectively blending warehouse and lake traits.

Introducing Delta Lake: Reliable Storage for the Lakehouse

Delta Lake is the open-source storage layer at the heart of many lakehouses. Originally developed at Databricks and later donated to the Linux Foundation, Delta Lake brings ACID transactions, schema enforcement, and performance optimizations to existing data lakes. In essence, Delta Lake is Parquet + a transaction log. Data is stored in standard columnar Parquet files, but every change (append/overwrite/update/delete) is recorded in a Delta transaction log (_delta_log directory). This log of JSON/checkpoint files enables ACID semantics, versioning (time travel), and efficient metadata management.

Some key facts about Delta Lake:

• Open Source & Community: Delta Lake has been used by thousands of companies and is a Linux Foundation project. Databricks announced that over 4,000 organizations were already processing exabytes of data on Delta Lake (as of 2019), and by 2022 they reported over 7,000 organizations using it globally. All of Delta Lake’s code is Apache-licensed (v2), including recent performance optimizations that Databricks open-sourced in Delta Lake 2.0.
• ACID Transactions: Delta Lake provides full ACID guarantees on data lakes. Every write (batch or streaming) is atomic – it fully succeeds or fails, so readers never see partial updates. This means failed jobs do not corrupt the table, and concurrent readers always see a consistent snapshot.
• Scalable Metadata: Delta can handle petabyte-scale tables with billions of files and partitions. Unlike a vanilla Parquet lake, Delta keeps track of files in the transaction log, avoiding costly cloud file-listing operations. Metadata lookups (for a query) scan the log, not the entire directory.
• Unified Batch & Streaming: Delta tables can be read and written by both batch and streaming jobs with exactly-once semantics. You can have streaming ingest continuously append data, while historical batch jobs backfill old data, all in the same table.
• Schema Enforcement/Evolution: By default, Delta ensures incoming data matches the table schema, preventing bad records from corrupting the dataset. It also supports safely evolving schemas over time.
• Time Travel (Versioning): Every change in a Delta table is versioned. You can query or even restore a table to a previous state (e.g. for audits, debugging, or rollbacks).
• DML Operations: Unlike raw data lakes, Delta supports SQL MERGE/UPDATE/DELETE on existing data, making it easy to comply with GDPR/CCPA (delete a user’s data) or correct errors in place.
• Open Format: Delta tables can be read by many engines (Spark, Presto, Trino, Hive, etc.) using the open Delta format. Databricks even created “Delta Share” as an open protocol to share Delta tables across organizations.

Note: Delta Lake is essentially an ACID transaction layer on Parquet data lakes. It adds reliability and SQL capabilities to your lake without locking you into a proprietary system.

Delta Lake vs. Raw Data Lake: Solving Data Lake Problems

Delta Lake was invented to fix the pain points of raw data lakes. Consider some concrete examples:

• Recovered from failed writes: In a raw lake, if a Spark job fails mid-write, you might end up with partial Parquet files. Engineers would have to manually clean up and re-run. Delta avoids this by writing log entries in a transactional way. As Databricks explains, “every operation performed on [the table] is atomic: it will either succeed completely or fail completely,” so the lake stays clean. You no longer need to manually reprocess broken pipelines – Delta keeps data consistent even after failures.
• Easy Updates and Deletes (GDPR compliance): Suppose a customer invokes the “right to be forgotten.” On a plain data lake, deleting all their records is cumbersome: you’d have to rewrite huge Parquet files and then clean up old ones. Delta makes this simple: since it supports SQL DELETE, you can issue DELETE FROM table WHERE customer_id = X, and Delta’s ACID engine will update the table cleanly. After the delete, running VACUUM removes the old files. This unified approach means auditors can verify compliance (via time travel) and pipelines aren’t broken.
• Schema Enforcement and Evolution: Raw lakes lack any guardrails on schema changes. If one job writes a new column or an unexpected type, downstream queries can silently break. Delta enforces the declared schema, rejecting bad data (or auto-evolving when enabled). This prevents many bugs. For example, a new field arriving without warning will cause an error, protecting other analysts. (Delta also supports the opposite: if you do want the new column, you can enable schema evolution to incorporate it safely.)
• Time Travel for Audits and Debugging: Every version of a Delta table is recorded. You can query table VERSION AS OF N or restore a table to an earlier snapshot. This is invaluable for audits or investigating when data changed. Raw lakes have no built-in versioning – you would need manual snapshots, which are error-prone. Delta’s version history lets you answer questions like “what did this report look like last week?” or “when was this error introduced?” without complex workarounds.
• Performance Optimizations: Delta Lake also includes capabilities like data skipping and file compaction (OPTIMIZE) to speed up queries. For example, Delta keeps min/max statistics on columns, so queries can skip irrelevant files entirely. It also provides an API (OPTIMIZE) to merge many small files into larger ones for faster scans. These features are either manual or unavailable in a raw Parquet lake.

In summary, Delta Lake solves raw lakehouse pain points by enforcing transactions, allowing DML, and providing higher-level management. As Databricks notes, Delta “makes your data lake behave like a database, without sacrificing scale”. The result is that data teams spend less time fixing corrupt data and more time extracting insights.

Delta Lake Architecture Under the Hood

Underneath, a Delta table is still just files on storage plus a log of changes. Here’s a simplified view of a Delta table directory:

```

hide

/my-delta-table/

├── _delta_log/

│ ├── 00000000000000000001.json

│ ├── 00000000000000000002.json

│ └── ...

├── part-0001-*.parquet

├── part-0002-*.parquet

└── ...

```

Each JSON log file in _delta_log records one atomic transaction (e.g. which Parquet files were added or removed). When Spark or another engine queries the table, Delta reads the latest log, then knows exactly which Parquet files make up the current version of the table. Because the log is linear and append-only, Delta can also reconstruct any past snapshot by replaying logs up to a given version (enabling time travel).

By storing metadata (schema, partition info, and file lists) in the log, Delta avoids expensive directory listing calls in cloud stores. Instead of scanning millions of files to know what to read, Delta simply reads the small JSON log. This makes queries on very large tables efficient and scalable.

In practice, users interact with Delta through high-level Spark/Python/SQL commands. For example, Apache Spark’s merge or SQL UPDATE/DELETE translate into Delta log appends. Delta’s APIs (Scala, Java, Python, Rust) also allow writing data directly to a Delta table from any compute engine. In this way, Delta Lake remains an open format: the table data is Parquet (readable by anyone), and the protocol (_delta_log) is documented as an open spec.

Note: Internally, Delta Lake is “Parquet + transaction logs”. This simple structure is the secret sauce that gives you ACID and versioning without changing the underlying storage.

Comparing Warehouse, Lake, and Lakehouse

This comparison highlights why modern architectures converge on lakehouses. Delta Lake tables can serve both analytical queries and machine learning workloads on the same data. For instance, an insurance company might feed raw sensor data into a Delta table (like a data lake), enforce schemas and clean it in a “silver” layer, and then run BI dashboards off that same table without duplication. Downstream teams can also train ML models on the latest data without moving it.

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges