Why architecture matters here
Iceberg fails on metadata bloat (no expiration), concurrent write conflict, and partition evolution gone wrong. Architecture matters because metadata lifecycle + partition strategy shape performance.
The architecture: every piece explained
The top strip is metadata. Data files parquet. Manifest files list data files with stats. Manifest list is the commit boundary. Metadata JSON tracks current snapshot.
The middle row is behavior. Snapshots point-in-time. Partition spec evolves. Hidden partitioning derives keys. Time travel reads AS OF.
The lower rows are ops. Compaction + rewrite clusters + resizes. Catalog — Hive/Glue/REST. Ops — retention + GC + concurrent writers.
End-to-end flow
End-to-end: streaming writer commits new data. Manifest list updated. Snapshot advanced. Analytical reader queries AS OF prior snapshot. Partition evolution changes partitioning without rewrite. Compaction bin-packs small files.