Why it matters
Every Impala operational issue traces to one of these daemons. Query slowness is Impalad. Slow metastore sync is Catalog. Slow cluster health changes are Statestore. Knowing the mapping is what makes ops fast.
The architecture
Impalad has three roles: query planner (when it receives a query), query coordinator (when it's the target of a query), and query executor (when running fragments for a query). Any Impalad can play any role. In practice, load-balanced client connections spread coordination load.
Statestore is a lightweight publish-subscribe service. It maintains topics (cluster membership, catalog updates) and pushes updates to subscribed Impalads on any change.
Catalog service watches the Hive Metastore for changes, caches full metadata locally, and pushes deltas to Impalads via the Statestore.
How it works end to end
On startup, an Impalad subscribes to statestore topics. It receives the current cluster membership and current catalog snapshot. It then registers itself in the membership topic so other Impalads know it exists.
When a query arrives at an Impalad, the daemon plans it using cached catalog, dispatches fragments to relevant Impalads (identified via membership), and coordinates result gathering.
Catalog changes flow: DDL runs on any Impalad → Impalad forwards to Catalog service → Catalog updates metastore → Catalog pushes delta to Statestore → Statestore broadcasts to all Impalads.
One binary, three roles, two runtimes
The first thing to internalise about Impala is that impalad, statestored and catalogd are three process types, not three tiers of a layered application. They ship from the same source tree, they speak Thrift to each other, and none of them owns durable state. The authoritative copies of everything live somewhere else: table definitions in the Hive Metastore, files and blocks in HDFS or an object store, Kudu rows in Kudu. Every Impala daemon is a cache or a scheduler over that truth. Kill any of them and you lose warm state and in-flight queries, never data.
The second thing is stranger and explains most of Impala's operational personality: an impalad is a C++ process with a JVM embedded inside it over JNI. The frontend - SQL parser, semantic analyser, cost-based planner, Hive Metastore client, Ranger authorization plugin - is Java, because that is where the Hadoop ecosystem's client libraries live and where a planner can be written quickly. The backend - scanners, hash tables, aggregation, sorting, the exchange layer, the buffer pool - is C++, because the data path cannot afford a garbage collector. A batch engine can absorb a two-second stop-the-world pause inside a forty-minute job. An interactive engine with a one-second budget cannot, and no amount of GC tuning makes that pause disappear at the ninety-ninth percentile.
So a single impalad process has two memory regimes that are accounted for separately, and confusing them is a recurring support ticket. The embedded JVM has an ordinary heap sized by the usual JVM flags, and it holds the planner's working set and the coordinator's catalog cache. The C++ side has its own explicitly tracked arena, capped by mem_limit and subdivided by per-query and per-operator trackers. A coordinator can die of Java heap exhaustion while its /memz page reports gigabytes of unused mem_limit headroom, because the pressure was in the other runtime entirely.
Control plane and data plane
Split the daemons by what they carry rather than by what they are called. statestored and catalogd are pure control plane: they never see a row of user data. Query rows only ever move between impalad processes. That separation is why the control plane can be small - one statestore and one catalog daemon serve a cluster of hundreds of executors - and why control-plane outages degrade freshness and scheduling rather than corrupting results.
statestored is a soft-state publish/subscribe broker. Subscribers register for topics, receive periodic deltas and heartbeats, and that is the whole contract. It is deliberately not a consensus system, holds nothing on disk, and rebuilds its topics from scratch when subscribers re-register after a restart. Three things ride those topics: cluster membership (which backends are alive, which feeds the scheduler), catalog updates, and admission-control state. The deep treatment is in the statestore architecture article.
catalogd is the single writer for metadata. It loads schema, partition and file metadata from the Hive Metastore, versions every change monotonically, and broadcasts deltas over the statestore so that no coordinator ever has to call the metastore at plan time. The versioning, the local-catalog mode, and the REFRESH versus INVALIDATE repair verbs are covered in the catalog and metadata plane article; treat it here simply as the reason planning is a cache lookup instead of an RPC storm.
impalad straddles both planes, and which halves it runs are flags, not separate builds. --is_coordinator enables the frontend, the catalog subscription and the client-facing endpoints. --is_executor enables fragment execution. Both true is the default and the right answer on a small cluster. Coordinator-only and executor-only are the large-cluster split. Both false is legal and useless.
The wire: ports and the debugging surface
The port map is worth memorising because it is also the topology map - it tells you exactly who is allowed to talk to whom.
impalad
21050 HiveServer2 Thrift - JDBC/ODBC clients, impala-shell
21000 Beeswax Thrift - legacy client protocol
22000 backend (be_port) - impalad to impalad: fragment RPC, data exchange
23000 statestore subscriber - inbound topic updates from statestored
25000 web UI / metrics
statestored
24000 statestore service - subscriber registration and heartbeats
25010 web UI
catalogd
26000 catalog service - coordinator to catalogd DDL and metadata RPC
25020 web UINote the asymmetry. Client traffic terminates only on coordinators (21050). Data traffic is entirely executor-to-executor on 22000 and never touches the control plane. The statestore pushes to subscribers on 23000 rather than being polled, so a network partition that blocks that one direction produces the classic symptom of daemons that look healthy locally but vanish from cluster membership.
The three web UIs are the actual debugging tool, not a nicety. On a coordinator, /queries lists running and recently completed queries with their profiles, /sessions shows connected clients, /memz breaks down the C++ memory trackers, /backends lists the executors this daemon believes are alive, and /threadz and /metrics cover the rest. On statestored, /subscribers and /topics answer "does the cluster agree on who exists". On catalogd, /catalog enumerates loaded objects and their versions. Because these pages expose full query text, they need authentication and a firewall in any environment with real data in it.
Why shared-nothing daemons hit latencies a batch engine cannot
The usual explanation for Impala's speed is C++ and code generation. Those matter, but they are not the structural reason. The structural reason is that nothing has to be created when a query arrives.
Account for what a container-based batch engine spends before it reads its first byte. It negotiates with a cluster resource manager for containers; the manager schedules against a heartbeat interval measured in seconds. Each granted container launches a JVM, which loads classes, and then runs interpreted until the JIT decides the loops are hot enough to compile. At the end, all of it is torn down so the next query can pay the same bill. Even with every warm-start trick applied, the floor is seconds, and it is a floor that does not move when the query is trivial.
Impala's daemons are already running, with a warm JVM in the frontend, a populated catalog cache, open file handles and a warm data cache. The first cost a query pays is parsing. Planning is arithmetic over cached statistics. Fragment dispatch is an RPC to processes that already exist. A query that touches one small partition can finish in the time a batch engine spends waiting for its first container.
Shared-nothing is the other half. Executors share no memory, no lock manager and no coordination state; each owns a disjoint set of scan ranges assigned by the coordinator, and the only inter-node traffic is the exchange operators that shuffle or broadcast rows between plan fragments. Adding capacity is starting a process that registers with the statestore. There is no rebalancing step and no global structure to update.
The bill for this comes due in two places. Provisioning is per-cluster, not per-query - the daemons are resident whether or not anyone is querying, which is why admission control has to arbitrate a fixed pool instead of a resource manager growing one. And there is no mid-query fault tolerance: with no materialised stage boundaries there is nothing to restart from, so losing an executor kills every query with a fragment on it. That is the correct trade for dashboards and the wrong one for eight-hour ETL, which is exactly the line drawn in the Hive article. The fragment-level mechanics of planning and execution are in query execution.
Why admission control lives inside the cluster
Impala's resource gating looks eccentric until you know that the obvious alternative was tried and abandoned. Early versions integrated with YARN through a broker daemon called Llama - Low Latency Application Master - which translated Impala's per-query resource estimates into YARN reservations. It did not survive, and the reasons are architectural rather than incidental.
A general-purpose resource manager is built around allocating a container and starting a process in it. Impala's processes are already started and must stay resident to be fast, so there is nothing for the manager to launch; the reservation degenerates into an accounting fiction. Worse, the round trip to ask for it costs hundreds of milliseconds to seconds - the entire latency budget of the queries Impala exists to serve. Paying a second of scheduling overhead to protect a 400ms query is a losing exchange.
So admission moved in-process. Coordinators evaluate each query's planned per-node memory against named resource pools with their own memory and concurrency budgets, then admit, queue with a timeout, or reject with a reason. Pool state is gossiped over the statestore alongside membership. The vestigial configuration names give the history away: pools are still defined in a fair-scheduler style allocation file, and the flag that points at the per-pool Impala settings is still --llama_site_path.
The design's honest weakness is that gossiped state is eventually consistent. Two coordinators admitting at the same instant both see stale pool usage and can jointly overshoot a pool's budget before the next update reconciles them. Impala accepts a brief, bounded overshoot in exchange for an admission decision that costs microseconds and never leaves the process. Understanding that is the difference between sizing pools with headroom and being surprised when a pool with a 200GB budget momentarily runs 230GB. What happens after admission - per-query mem_limit, operator reservations, spilling - belongs to memory limits and spill to disk.
Sizing the roles: dedicated coordinators and executor groups
Below roughly twenty nodes, run every impalad as both coordinator and executor and stop thinking about it. Past that, the coordinator role becomes the scaling constraint, for four compounding reasons: it holds a full catalog cache in the JVM heap, it runs the planner for every query it accepts, it is the single point where all final result rows converge for merging, and it pins the result set and profile of every running query it owns. None of those costs shrink when you add executors. Several of them grow.
The symptom is unmistakable once you know it: in a hundred-node cluster, a handful of nodes are pegged while the rest idle, planning latency creeps up, and coordinator JVM heap pressure appears with no corresponding mem_limit pressure. The fix is the dedicated-coordinator topology - two or three coordinator-only daemons behind a load balancer, everything else executor-only.
# dedicated coordinator (2-3 of these, behind the load balancer)
impalad --is_coordinator=true --is_executor=false \
--state_store_host=ss.internal --catalog_service_host=cat.internal \
--mem_limit=64G
# executor-only worker (the other 97 nodes)
impalad --is_coordinator=false --is_executor=true \
--state_store_host=ss.internal \
--mem_limit=220GIt is also a metadata-plane optimisation, since only coordinators subscribe to the catalog topic - see the catalog article for what that fan-out costs.
The load balancer in front of 21050 has one non-obvious requirement: HS2 sessions are stateful. Session variables, temporary state, the query's result set and its profile all live on the coordinator that admitted the query, so balancing must be connection-level with source affinity and a generous idle timeout. Round-robin at any finer granularity produces the classic complaint that a query "disappears" when the client reconnects to a different coordinator. Coordinator memory pinned by undrained result sets is what result spooling exists to release.
Executor groups take the split one step further: executors are tagged into named groups and a query is scheduled entirely within one group. That gives workload isolation without separate clusters, and it makes horizontal scaling a matter of adding or removing whole groups rather than individual nodes - the unit that elastic deployments actually want.
Failure modes, mapped to the daemon that owns them
Every Impala incident traces to one of the three daemons, and triage speed comes from having that mapping memorised rather than from reading logs in order.
| Symptom | Owner | Mechanism | First move |
|---|---|---|---|
| Memory limit exceeded, one operator named | executor impalad | Cardinality misestimate blows past the operator's reservation | COMPUTE STATS on the join keys; check estimate vs actual in the profile |
| Queries sit in the queue and time out | coordinator (admission) | Pool budget too small, or one hung query holding the pool | Pool sizing; find the long-running query holding the memory |
| Table or partition not found after a Spark or Hive write | coordinator catalog cache | External writer changed the metastore; the cache has not been told | REFRESH the table; enable metastore event polling so it is automatic |
| catalogd restart takes many minutes; heap climbs | catalogd | Working set of partitions and file descriptors exceeds the heap | Local-catalog mode; fix partition counts and small files |
| Scan ranges assigned to a node that is gone | statestored | Stale membership, usually a network or heartbeat problem | Check /subscribers on the statestore web UI |
| Three nodes hot, ninety-seven idle | topology | Coordinator role is the bottleneck, not the cluster | Split into dedicated coordinators and executors |
| Every query on a node dies when it reboots | by design | No mid-query fault tolerance in a pipelined MPP engine | Retry at the client, or use transparent query retry |
Two habits make the table almost unnecessary. First, read the query profile before anything else - it names the fragment instance, the host and the operator, which collapses three of these rows immediately. Second, remember what each daemon losing state actually costs. A statestore restart costs freshness for a few seconds. A catalogd restart costs a cold metadata load. An executor restart costs the queries that were running on it. None of them costs data, which is why the safest response to a confusing Impala cluster is usually a rolling restart, and why that same instinct would be catastrophic against a NameNode.