Why architecture matters here
The architecture matters because the naive alternatives to disseminating cluster state are all worse for an interactive analytics engine. A strongly-consistent coordination service (a Paxos/Raft cluster) would make every membership and metadata change a consensus operation — durable, ordered, and correct, but with latency and operational weight that Impala's use case does not need. Impala's shared state is soft: a daemon that briefly does not know about a just-created table can find out a second later, and a coordinator that briefly thinks a dead executor is alive simply fails a fragment and retries elsewhere. Paying the cost of consensus for data that tolerates seconds of staleness would be over-engineering. The statestore's lightweight pub/sub matches the actual consistency requirement.
Failure detection is the first job, and it is why membership must be centralized rather than gossiped ad hoc. Each subscriber exchanges periodic heartbeats with the statestore. When a daemon's heartbeats stop for long enough, the statestore declares it failed and publishes the updated membership to every subscriber, so coordinators quickly stop routing query fragments to a dead executor. A single, consistent authority for 'who is alive' — even a soft-state one — avoids the split-brain confusion of every daemon independently guessing at its peers' health.
Metadata propagation is the second job, and decoupling it through the statestore is what keeps queries fast. Rather than every daemon polling a metadata store or querying the Hive Metastore on every query, the catalog service publishes metadata changes as topic updates that the statestore fans out to all daemons. A daemon therefore usually has the metadata it needs already in memory, cached and kept fresh by the update stream, so query planning does not pay a metadata round-trip. The statestore turns 'everyone pulls metadata constantly' into 'changes are pushed once and cached everywhere'.
Finally, the soft-state, off-the-hot-path design is what makes the statestore operationally forgiving, and that is a feature, not an accident. Because the statestore is not consulted during query execution — daemons act on their locally-cached topic state — a statestore that is slow or briefly down does not stall running queries. And because its state is reconstructable, restarting it is a routine, low-risk operation rather than a cluster-threatening event. The architecture deliberately keeps the one central component off the critical path and disposable, which is precisely why a single statestore is acceptable in a system that otherwise avoids single points of failure.
The architecture: every piece explained
statestored is the central broker process, and there is normally one per cluster. It maintains, in memory, the set of topics and the set of subscribers, and it does the fan-out: when a topic changes, it pushes the change to every subscriber registered for that topic. It persists nothing and it is not the origin of any data — it is a relay with failure detection bolted on. Its simplicity is the point: less to go wrong in the one component every daemon depends on.
Topics are named channels of related updates. The two that matter are the membership topic, which carries the set of live impalad daemons, and the catalog topic, which carries metadata objects — databases, tables, partitions, functions — and their versions. Topic entries are versioned and updates are incremental: a publisher sends only the deltas since the last version, and subscribers apply them to their local copy, so a busy catalog does not require re-sending the whole metadata universe on every change.
Subscribers are the daemons that register interest. Every impalad subscribes to membership and catalog topics; the catalogd both publishes to and interacts with the catalog topic. A subscriber, on registering, gets the current topic state and thereafter receives deltas. Each maintains a local, in-memory copy of the topics it cares about, which is what it actually reads during planning and scheduling — the statestore is a delivery mechanism, not a lookup service the daemons call synchronously.
Heartbeats serve two purposes: they carry topic updates to subscribers, and their absence signals failure. The statestore sends periodic keepalive/topic-update messages; if a subscriber stops acknowledging within a threshold, the statestore marks it dead and publishes the membership change. The catalogd is a separate but closely-related daemon: it is the actual authority for catalog metadata (it talks to the Hive Metastore and HDFS), and it uses the statestore as its transport to broadcast metadata changes to all impalads. It is important not to conflate them — the catalogd owns and computes metadata; the statestore merely distributes it, alongside membership. Together — the statestored broker, versioned incremental topics, the subscriber model with local caches, heartbeat-based failure detection, and the catalogd as the metadata authority riding the bus — these form the complete architecture, and the soft-state principle ties them together: everything the statestore holds can be rebuilt from its subscribers.
End-to-end flow
Trace a daemon failure. Three impalad executors and one coordinator are running, all subscribed to the membership topic. One executor's host loses power. It stops sending heartbeats to the statestore. After the configured number of missed heartbeats, the statestore declares that executor failed and publishes an updated membership topic — the dead daemon removed — to all remaining subscribers. The coordinator applies the delta to its local membership copy and, from that point, schedules query fragments only across the surviving executors. A query that had a fragment on the dead node fails and is retried on the reduced cluster. The whole detect-and-disseminate cycle takes seconds, and no central authority had to be consulted synchronously by any query — coordinators simply read their locally-updated membership.
Now trace a metadata change. A user runs CREATE TABLE sales.orders ... against one coordinator. That coordinator asks the catalogd to perform the DDL; the catalogd creates the table in the Hive Metastore, updates its own authoritative catalog, and publishes the new/changed catalog objects as an incremental update to the catalog topic. The statestore fans that delta out to every subscribed impalad, each of which applies it to its local catalog cache. Within a short window, every daemon in the cluster knows about sales.orders and can plan queries against it — without any daemon having queried the Metastore directly. The statestore turned one DDL into a single broadcast rather than N independent metadata fetches.
Consider the statestore restarting mid-operation — the soft-state payoff. Suppose an operator restarts statestored. For a brief moment it has no topics and no subscribers. The running impalad daemons, acting on their already-cached membership and catalog state, keep planning and executing queries — the statestore is not on the query hot path, so in-flight work is unaffected. As the daemons detect the statestore is back, they re-register as subscribers and re-publish their state; the catalogd re-publishes the catalog topic. Within seconds the statestore's in-memory view is fully reconstructed from the live cluster, and delta delivery resumes. Nothing was persisted and nothing needed to be, because every piece of state the statestore carries is owned by some daemon that can supply it again.
Finally, the steady state. Heartbeats flow continuously between the statestore and every subscriber, doubling as the topic-update channel. Membership is stable, so most heartbeats carry no membership delta. The catalog changes only when DDL or a metadata refresh happens, so most heartbeats carry no catalog delta either — the incremental, versioned design means quiet periods cost almost nothing. When a change does occur, it is published once and propagated everywhere. The cluster runs with each daemon holding a fresh-enough local view, the statestore quietly relaying changes and watching for the silence that means a daemon has died, and the whole coordination fabric staying off the latency-critical path of the queries it serves.