Delta Lake Architecture in Depth

Why architecture matters here

Delta Lake architecture matters because "just Parquet on S3" gives you data but not ACID. Concurrent writers corrupt each other; readers see partial writes; DELETE and UPDATE are hard. Delta's transaction log solves all of these while keeping data in open Parquet.

Cost is largely inherited from S3 + Parquet. Delta's overhead is small — the transaction log adds JSON metadata, and VACUUM eventually removes old files.

Reliability comes from atomic commits. A commit either succeeds fully (log entry written) or fails (log entry not written). Optimistic concurrency handles concurrent writers.

The architecture: every piece explained

Walk the diagram top to bottom.

Writer (Spark). Application code writing to a Delta table. Standard Spark writes go through Delta's format.

Delta Transaction Log. Directory _delta_log/ under the table. Numbered JSON files: 000000.json, 000001.json, etc. Each represents a commit.

Reader. Any process reading the table. Reads the log to determine which Parquet files are active for a version.

Parquet Data Files. Immutable. New files added on write; old ones marked removed in log but not deleted immediately.

Checkpoints. Every N (default 10) commits, a checkpoint Parquet file consolidates the log so readers don't have to replay all JSONs.

OPTIMIZE + Z-ORDER. OPTIMIZE compacts small files into larger ones. Z-ORDER sorts rows within files by specified columns for skip-based reads.

Time Travel. SELECT * FROM table VERSION AS OF 42 or TIMESTAMP AS OF '2026-05-01'. Reader consults log to find files active then.

MERGE / UPSERT. SQL MERGE INTO ... WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT. Atomic upserts.

Streaming Source + Sink. Delta table as streaming source (read new commits incrementally) or sink (append + exactly-once).

VACUUM. Removes data files older than retention window (default 7 days). Reclaims storage. Blocks time travel beyond window.

End-to-end UPSERT + time travel flow

Trace an UPSERT. Application has a batch of updates to a large customer table. Runs MERGE INTO customers USING updates ON id = id.

Delta reads the log to find current active files. Reads relevant Parquet files, identifies matched vs unmatched rows.

Writes new Parquet files with combined data. Prepares a commit: mark old files as removed, add new files.

Attempts atomic commit: write 000042.json to _delta_log/. Optimistic concurrency check succeeds (no conflicting writer). Commit visible.

Reader queries. Reads latest checkpoint + log entries up to 42. Sees new files, skips removed. Query returns updated data.

Time travel: SELECT * FROM customers VERSION AS OF 41. Reader replays log to version 41. Returns pre-merge state.

Six months later: OPTIMIZE ZORDER BY (region, country). Delta rewrites small files into large sorted files. New commit.

VACUUM: after 7 days, VACUUM deletes physical files that are no longer active. Time travel beyond that window fails.

Streaming: another job reads Delta table as source. Sees only new commits after its checkpoint. Exactly-once processing.

Inside _delta_log: actions, not state

The log does not store the table’s state. It stores an ordered sequence of actions, and state is the fold over them. A commit is one JSON file named with a zero-padded 20-digit version, _delta_log/00000000000000000042.json, one JSON object per line:

• protocol - the minimum reader/writer versions, or named table features, a client needs to touch this table.
• metaData - schema, partition columns, table properties.
• add - a file joining the table: path, partition values, size, a dataChange flag, a stats blob.
• remove - a tombstone. The Parquet file still exists on storage; only the log says it left the table.
• txn - application id plus monotonic version, for idempotent streaming writes.
• commitInfo - provenance; what DESCRIBE HISTORY reads.

Since adds and removes are the only way the file set changes, every operation - append, overwrite, DELETE, MERGE, OPTIMIZE - reduces to one primitive: write Parquet, then commit adds and removes atomically. No data file is mutated in place.

Checkpoints and how a reader rebuilds a snapshot

Replaying every commit from version 0 would make snapshot construction O(commits), fatal for a table committing once a minute. So every delta.checkpointInterval commits (10 by default) a writer materialises the reconstructed action set into a Parquet checkpoint and updates the tiny _last_checkpoint pointer; big tables split it into parts that read in parallel.

_delta_log/
  ...040.checkpoint.parquet   <- state at v40
  ...041.json                 <- deltas since
  ...042.json
  _last_checkpoint            <- {"version":40}

A reader loads the checkpoint, applies the higher-numbered JSON commits, and holds an in-memory list of live files and their statistics. That list is the snapshot. So planning cost tracks commits-since-checkpoint plus live file count, not table age - and because the driver holds that list in memory, a table with millions of live files makes planning itself the bottleneck. That is the real argument for compaction.

Atomic commit and optimistic concurrency

Everything rests on one primitive: create version N of the log if and only if nobody else has. On HDFS and ADLS that is an atomic rename. S3 was long the awkward case - PutObject is last-writer-wins, so two writers can both believe they created ...042.json and one commit silently vanishes. Delta’s answer is a pluggable LogStore; the default is safe only because a single driver serialises its own commits. Multiple independent writers on S3 need a multi-cluster log store - the DynamoDB-backed one, or one using S3 conditional writes. Skip it and you get silent commit loss, not an error.

Above that sits optimistic concurrency: read snapshot R, do the work, try to commit R+1. If that version exists, read the missed commits and test for real conflict; if none, rebase and retry. Conflicts surface as ConcurrentAppendException (files appeared that your predicate would have read - the common one), ConcurrentDeleteReadException, or MetadataChangedException. Granularity is the file set your predicate implies, so the fix for append conflicts is a narrower predicate: a MERGE whose ON clause names the partition column is provably disjoint from writes to other partitions.

Snapshot isolation, time travel, and retention

Readers get snapshot isolation for free: a query pins a version at planning time and that file list is immutable, so a ten-minute scan is unaffected by a hundred commits landing under it. Time travel is the same mechanism with the version named explicitly.

SELECT * FROM orders VERSION AS OF 118;
SELECT * FROM orders TIMESTAMP AS OF '2026-07-01 00:00:00';
RESTORE TABLE orders TO VERSION AS OF 118;
VACUUM orders RETAIN 168 HOURS DRY RUN;   -- list, delete nothing

RESTORE is not a rollback of history: it appends a new commit reproducing the old file set. The boundary on all of this is retention. Tombstoned files survive until VACUUM deletes those older than delta.deletedFileRetentionDuration (7 days); log history survives delta.logRetentionDuration (30 days). The readable horizon is the smaller of the two.

That 7-day default is a safety margin for in-flight work: a long query still reading files a concurrent OPTIMIZE tombstoned, or a writer with staged files not yet committed. Cutting retention to zero is the classic way to kill a running job with FileNotFoundException, which is why Delta guards it behind a config flag.

Schema enforcement, evolution, and protocol versions

Schema lives in the metaData action, so it is transactional. A write whose schema is incompatible is rejected before any data lands - Delta will not silently widen an int to a string or absorb an unknown column.

(df.write.format("delta").mode("append")
   .option("mergeSchema", "true")       # additive: new columns allowed
   .save(path))

Additive evolution is cheap because it rewrites only metadata; overwriteSchema is the destructive escape hatch. Rename and drop are harder, because Parquet addresses columns by name. Column mapping (delta.columnMapping.mode = 'name') puts an indirection between logical name and physical field id, making RENAME and DROP COLUMN metadata-only.

Such features bump the protocol action, and older readers then refuse the table rather than misread it - so enabling column mapping or deletion vectors is a compatibility decision across every engine touching that table.

Row-level writes: copy-on-write vs deletion vectors

Parquet is immutable, so classic DELETE, UPDATE and MERGE are copy-on-write: locate the files holding affected rows, rewrite them with the change applied, commit adds plus removes. MERGE needs two passes: find the matching files, then rewrite them.

The cost model is worth internalising: deleting one row from a 1 GB file rewrites 1 GB, so an erasure job touching 5,000 scattered rows in a 2 TB table can rewrite much of it. Write amplification, not row count, is what makes these jobs slow.

Deletion vectors change the trade. With delta.enableDeletionVectors = true, a DELETE writes a compact bitmap of deleted row positions beside the untouched Parquet file. Deletes become nearly free, paid for with merge-on-read work: every scan applies the vector, and updates land as new rows plus a vector entry. Vectors accumulate, so materialise them periodically - REORG TABLE ... APPLY (PURGE) rewrites the files with deletions applied, which is also what you run before claiming a row is gone from storage.

File layout: OPTIMIZE, statistics, and Z-ordering

Every add carries a stats blob: row count, per-column min, max and null count. Because that sits in the log, the planner eliminates whole files from metadata alone, with no footer reads. That is data skipping, distinct from directory-level partition pruning, which the neighbouring partition pruning article covers. Stats cover the first delta.dataSkippingNumIndexedCols columns (32 by default), so schema column order is a tuning knob: a selective filter column at position 60 is invisible to skipping.

OPTIMIZE events WHERE event_date >= '2026-07-01'
  ZORDER BY (user_id, device_id);

OPTIMIZE is a pure layout change: its commit carries dataChange = false, so streaming readers ignore it instead of re-emitting rewritten rows. Z-ORDER interleaves the bits of several columns so min/max ranges stay tight in more than one dimension - a plain sort makes the first column skip beautifully and the second not at all. Two columns is the sweet spot; by four, each dimension is diluted enough that the rewrite rarely pays for itself.

Streaming source and sink semantics

An ordered append-only commit log is exactly what a streaming source needs, so a Delta table is one natively. The offset is a version plus an index within it, and each trigger reads the files added by later commits. Rate control is maxFilesPerTrigger or maxBytesPerTrigger; startingVersion begins mid-history.

(spark.readStream.format("delta")
   .option("maxBytesPerTrigger", "2g")
   .option("skipChangeCommits", "true")   # tolerate upstream MERGE
   .load("/lake/events"))

The source contract is append-only. If upstream runs UPDATE or MERGE the stream fails rather than guess; skipChangeCommits ignores those commits. If you want the changes, enable the change data feed and read readChangeFeed, which yields explicit insert, update and delete rows. On the sink side exactly-once comes from the txn action: each commit is stamped with application id and batch version, so a replayed micro-batch is recognised and dropped. Watermarks and state are covered in Structured Streaming state.

Production failure modes

Small files. A job triggering every 30 seconds commits roughly 2,900 files a day, each an add the driver holds in memory. Degradation is gradual - a 40-second query becomes a 12-minute one - and scheduled OPTIMIZE belongs in the design, not the post-mortem.

Concurrent-write conflicts. Ten hourly MERGE jobs into one unpartitioned table serialise into a retry storm. Partition on the dimension your writers separate on.

Log bloat. Millions of tiny commits make checkpoints expensive and every reader pays to load them. One commit a minute is fine; thirty is not.

Delta vs Iceberg vs Hudi: different bets on metadata

All three give ACID over object storage with open file formats; they diverge in how metadata is structured. Delta is a flat linear log plus checkpoints with statistics inline, so a snapshot is a fold over recent commits and atomicity is pushed down to storage - hence the S3 multi-writer caveat above. Iceberg is a metadata tree: a snapshot points at a manifest list pointing at manifests, and atomicity is delegated to a catalog swapping one pointer. The indirection buys manifest-level pruning and hidden partitioning. Hudi centres on a timeline plus a record-level index and makes copy-on-write versus merge-on-read a table type, which is why it appears in high-rate CDC ingestion. Short version: Delta for Spark-centric stacks wanting the simplest mental model, Iceberg for engine neutrality, Hudi when key-level upsert throughput dominates.