Why architecture matters here
The data cache is architectural because it directly addresses the defining cost of disaggregated storage: the network hop between compute and data. Separating storage from compute is what gives cloud analytics its elasticity — you can spin compute up and down without moving petabytes of data, and you pay for storage and compute independently — but the price is that data locality, the thing that made colocated Hadoop fast, is gone. Every scan is a remote read. A caching layer on local fast storage is how you buy back locality selectively, for the data that is actually hot, without giving up the elasticity of disaggregation. It is not a minor optimization; it is the mechanism that makes the disaggregated model performant enough to compete with the colocated one for interactive analytics.
The reason the cache is keyed by file and byte offset rather than by row or by query is that it operates at the storage layer, below any understanding of what the bytes mean. Impala reads columnar files in ranges — a Parquet row group's column chunk is a contiguous byte range in a file — and those ranges are exactly what repeat across queries that touch the same table. Caching at the file-offset granularity means the cache is oblivious to schema, predicates, and query shape; it just remembers 'these bytes of this file are on local disk', which is the most reusable unit possible because the same physical bytes are re-read by any query that scans that file, regardless of what those queries compute. This layer-of-abstraction choice is what lets one simple cache accelerate wildly different queries over the same data.
Immutability is the property that turns caching from a hard problem into an easy one, and it is worth dwelling on why. The reason most caches are dangerous is coherence: when the underlying data can change, a cached copy can become wrong, and keeping caches correct requires invalidation protocols that are notoriously hard to get right and a frequent source of subtle bugs. The Impala data cache sidesteps this entirely because the objects it caches — immutable data files — can never change their contents. A byte range at a given offset of a given file is write-once; there is no version of it that differs from the cached one. When the table's data changes, it changes by producing new files with new names, which have their own cache entries, while the old files (and their cache entries) are simply no longer referenced by any query and eventually age out. The cache never has to detect staleness because staleness is impossible by construction.
There is a deeper architectural dependency that decides whether the cache works at all, and it is the least obvious piece: the cache's effectiveness is a property of the scheduler, not just the cache. Because each daemon has its own independent local cache, a cached byte range only helps if the query fragment that needs those bytes is scheduled onto the same daemon that cached them. If the scheduler sends scans of a given file to a different daemon each time, every daemon suffers a cold miss and fetches the same bytes from the remote store, and the aggregate cache hit ratio collapses even though the data is technically 'cached' somewhere. This is why Impala's scheduler uses consistent affinity — it deterministically maps a given file (or file split) to a preferred daemon, so repeated scans of that file land on the daemon that already holds its bytes. The cache and the scheduler are therefore a single co-designed system: a per-node cache without affinity is nearly useless, and affinity without a cache is pointless. Understanding this coupling is the difference between an operator who enables the cache and gets little, and one who understands that the hit ratio depends on scan placement staying stable across queries and across the cluster's scaling events.
The architecture: every piece explained
Start with the request. A query fragment running on an Impala daemon needs to scan a byte range of a data file — for a columnar format, the column chunks its query touches within some row groups. Before going to the network, it does a cache lookup keyed by the file identity plus the offset and length of the range it wants. That key is exact: the cache answers 'do I have precisely these bytes of precisely this file on local disk?'. The local NVMe cache is a fixed-size region of fast local storage that each daemon manages independently — it is not shared across the cluster; every daemon caches what it reads. Behind it all sits the remote store — S3, ADLS, or remote HDFS — the durable but slow and distant source of truth.
The middle row is the read-through path. On a cache hit, the requested bytes are read straight from local SSD and handed to the scan, and no network request is made — this is the fast path the whole design exists to hit. On a cache miss, the daemon fetches the range from the remote store, returns it to the query so the scan is not blocked, and inserts a copy into the local cache so the next scan of that range hits. Because the cache is a fixed size, insertions eventually require LRU eviction: when the cache is full, the least-recently-used ranges are evicted to make room, so the cache naturally retains the hot working set and lets cold data fall out. The scheduler affinity is the piece that makes hits likely: Impala's scheduler maps a given file split to a preferred daemon consistently, so repeated scans of the same file are routed to the daemon that already cached it.
The bottom row holds the two properties that make the cache trustworthy and observable. Consistency is free because the cached files are immutable: a byte range's contents never change, so a cache hit can never return stale data and no invalidation is needed. Metrics are how you run the cache: the hit ratio (fraction of read bytes served from cache), the bytes cached and served, and the eviction rate together tell you whether the cache is sized right and whether affinity is working. A high hit ratio means the cache is doing its job; a low one with high evictions means the hot set does not fit; a low one with low evictions usually means affinity is broken and scans are scattering across daemons.
The ops strip distills the discipline. Size the cache to the hot working set — big enough that the repeatedly-scanned data fits, since a cache smaller than the hot set thrashes and helps little. Preserve scheduler affinity so a given file keeps landing on the daemon that cached it, especially across cluster scaling. Watch the hit ratio as the primary health signal. And isolate the cache's disk from other local I/O — particularly query spill — so that caching and spilling do not contend for the same NVMe bandwidth and undermine each other.
End-to-end flow
Trace a dashboard query that runs every few minutes against the same fact table. The first time it runs, the caches are cold. Each daemon's scan fragments look up their file-offset ranges, miss, fetch the column chunks from S3 across the network, return them to the query, and insert copies into their local NVMe caches. This first run pays the full remote-read cost and is the slowest it will ever be. But now the hot column chunks of that fact table's files are sitting on local SSD, distributed across the daemons according to scheduler affinity — each daemon holds the ranges for the file splits it was assigned.
The dashboard refreshes. The identical query runs again, and the scheduler, using consistent affinity, routes each file split to the same daemon that scanned it last time. Every fragment's cache lookup now hits: the column chunks are read from local NVMe at a fraction of the latency and none of the network cost of the object store. The query returns dramatically faster, the object-store request rate drops to near zero for this table, and — on a metered object store — the request-and-egress bill for the repeated scans vanishes. Because analytical dashboards re-scan the same hot tables constantly, the steady state is overwhelmingly cache hits, which is exactly the workload the cache is tuned for.
Consider what happens when a new day's data lands. An ETL job writes new Parquet files for today's partition into the object store. The dashboard query, now covering today, scans both the cached older files and the brand-new files. The new files are not in any cache, so their ranges miss and are fetched and inserted — but crucially, the old files' cached ranges are still valid and still hit, because those files were never modified; the new data arrived as new files, not as edits to existing ones. There is no invalidation to perform and no risk of serving stale bytes: immutability means the cache simply grows to include the new files' hot ranges and, under LRU, eventually evicts whatever has gone cold. The consistency question that plagues mutable-data caches never arises.
Now the failure that quietly kills the benefit: broken affinity. Suppose the cluster autoscaled — daemons were added — and the mapping of file splits to daemons shifted so that scans of a file that RS-A had cached are now sent to RS-B. RS-B has a cold cache for those ranges, so it misses and re-fetches from S3, while RS-A's cached copy sits unused and eventually evicts. If this scattering is widespread, the cluster's hit ratio collapses even though the aggregate cache is full of the right data — it is just cached on the wrong daemons relative to where scans are now landing. The query slows back toward cold-cache speeds and the object-store read rate climbs, all without any error. The operator watching only 'is the cache enabled?' sees nothing wrong; the operator watching the hit ratio sees it crater and knows to investigate scan placement. This is the concrete reason the cache and scheduler must be understood as one system: the cache's value is realized only when scans keep landing where their bytes already live, and any event that reshuffles that placement — scaling, daemon restarts, changes to scheduling — can erase the benefit until affinity re-stabilizes.