Why architecture matters here
Delta Lake architecture matters because "just Parquet on S3" gives you data but not ACID. Concurrent writers corrupt each other; readers see partial writes; DELETE and UPDATE are hard. Delta's transaction log solves all of these while keeping data in open Parquet.
Cost is largely inherited from S3 + Parquet. Delta's overhead is small — the transaction log adds JSON metadata, and VACUUM eventually removes old files.
Reliability comes from atomic commits. A commit either succeeds fully (log entry written) or fails (log entry not written). Optimistic concurrency handles concurrent writers.
The architecture: every piece explained
Walk the diagram top to bottom.
Writer (Spark). Application code writing to a Delta table. Standard Spark writes go through Delta's format.
Delta Transaction Log. Directory _delta_log/ under the table. Numbered JSON files: 000000.json, 000001.json, etc. Each represents a commit.
Reader. Any process reading the table. Reads the log to determine which Parquet files are active for a version.
Parquet Data Files. Immutable. New files added on write; old ones marked removed in log but not deleted immediately.
Checkpoints. Every N (default 10) commits, a checkpoint Parquet file consolidates the log so readers don't have to replay all JSONs.
OPTIMIZE + Z-ORDER. OPTIMIZE compacts small files into larger ones. Z-ORDER sorts rows within files by specified columns for skip-based reads.
Time Travel. SELECT * FROM table VERSION AS OF 42 or TIMESTAMP AS OF '2026-05-01'. Reader consults log to find files active then.
MERGE / UPSERT. SQL MERGE INTO ... WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT. Atomic upserts.
Streaming Source + Sink. Delta table as streaming source (read new commits incrementally) or sink (append + exactly-once).
VACUUM. Removes data files older than retention window (default 7 days). Reclaims storage. Blocks time travel beyond window.
End-to-end UPSERT + time travel flow
Trace an UPSERT. Application has a batch of updates to a large customer table. Runs MERGE INTO customers USING updates ON id = id.
Delta reads the log to find current active files. Reads relevant Parquet files, identifies matched vs unmatched rows.
Writes new Parquet files with combined data. Prepares a commit: mark old files as removed, add new files.
Attempts atomic commit: write 000042.json to _delta_log/. Optimistic concurrency check succeeds (no conflicting writer). Commit visible.
Reader queries. Reads latest checkpoint + log entries up to 42. Sees new files, skips removed. Query returns updated data.
Time travel: SELECT * FROM customers VERSION AS OF 41. Reader replays log to version 41. Returns pre-merge state.
Six months later: OPTIMIZE ZORDER BY (region, country). Delta rewrites small files into large sorted files. New commit.
VACUUM: after 7 days, VACUUM deletes physical files that are no longer active. Time travel beyond that window fails.
Streaming: another job reads Delta table as source. Sees only new commits after its checkpoint. Exactly-once processing.