HiveServer2 Clustering, in depth: ZooKeeper discovery, load balancers, active/passive, sizing and rolling restarts

What a single HS2 keeps to itself

Before choosing a topology, list what an HS2 keeps in its own memory, because none of it moves to another node when a server dies.

• Sessions, including every SET variable, the current database, added JARs and temporary functions.
• Operation handles for running and finished queries, and the result sets clients are still fetching.
• Temporary tables, which are scoped to the session and vanish with it.
• Tez sessions: the YARN application masters an HS2 starts or keeps in a pool to run queries quickly.
• Delegation tokens and Kerberos state for the users it is serving or impersonating.

What is shared lives below HS2: table definitions and statistics in the metastore, data and scratch directories in HDFS or object storage, and compute in YARN or LLAP. That split gives the defining property of HS2 clustering: you can add or lose servers without losing data or metadata, but a lost server takes its sessions and in-flight queries with it. Clients must reconnect and re-run. There is no transparent session failover.

ZooKeeper dynamic service discovery

The built-in clustering mechanism is dynamic service discovery through ZooKeeper. Each HS2 is configured with the same namespace and, at startup, creates an ephemeral sequential znode describing itself.

<!-- hive-site.xml on every HS2 node -->
<property><name>hive.server2.support.dynamic.service.discovery</name><value>true</value></property>
<property><name>hive.server2.zookeeper.namespace</name><value>hiveserver2</value></property>
<property><name>hive.zookeeper.quorum</name><value>zk1:2181,zk2:2181,zk3:2181</value></property>

A registered node appears under the namespace with a name like serverUri=hs2-a.example.com:10000;version=3.1.3;sequence=0000000042. Because the node is ephemeral, it disappears when the HS2's ZooKeeper session ends, whether through clean shutdown, crash or a network partition long enough to expire the session.

Clients do not name a server. They name the ZooKeeper ensemble and the namespace:

jdbc:hive2://zk1:2181,zk2:2181,zk3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

# HTTP transport through the same discovery
jdbc:hive2://zk1:2181,zk2:2181,zk3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=http;httpPath=cliservice

The driver reads the children of the namespace, picks one at random and connects to it directly. If that connection fails, it tries another registered server. Random choice spreads new sessions roughly evenly, but it knows nothing about load, so a node with forty heavy queries gets new sessions as often as an idle one. In newer drivers the HS2 also publishes its connection settings in ZooKeeper so clients pick up transport and security parameters from the registry; check your driver's documentation for what it reads.

Two subtleties matter. The ephemeral node lives as long as the ZooKeeper session, not as long as the server is healthy, so an HS2 stuck in a long garbage collection pause or with exhausted threads stays registered and keeps receiving new clients. And ODBC drivers from third parties implement discovery themselves, so test each BI tool you support, not just Beeline.

Load balancers as the alternative

The alternative is a load balancer in front of the pool, with clients pointing at one virtual hostname. This suits environments where clients cannot reach ZooKeeper, such as BI tools outside the cluster network, or where you want health checks that understand load.

• Use HTTP transport (hive.server2.transport.mode=http) for load balancers that work at layer 7, or binary transport with plain TCP balancing. Either way the session is bound to one server, so the balancer must keep a client on the same backend: source-IP affinity for TCP, or cookie stickiness for HTTP, where HS2 can issue its own auth cookie so it does not re-authenticate every request.
• Kerberos needs care. A client asks the KDC for a ticket to the hostname it connects to, which is now the balancer's name. Every HS2 must therefore hold a key for a service principal matching that name, typically by adding the balancer principal to each server's keytab and configuring the server principal accordingly. Distributions document this differently, so follow your vendor's guide; Hive security with Kerberos and Ranger covers the surrounding setup.
• Health checks should do more than open a TCP port. A port check passes on a server whose worker pool is exhausted. HS2 exposes a web UI with metrics; checking that it responds and that open sessions are below a threshold catches far more.

Many clusters use both: ZooKeeper discovery for internal jobs and an Apache Knox or load-balancer endpoint for external tools.

Active/passive mode

Hive 3 added an active/passive mode, enabled with hive.server2.active.passive.ha.enable, which registers instances in a separate ZooKeeper namespace and elects one leader that receives all connections while the others stand by. It exists mainly for HiveServer2 Interactive in LLAP deployments, where one server coordinates the LLAP daemons and their workload management, so having two active coordinators would be wrong. Cloudera Data Warehouse offers a similar active/passive option for its HS2 pods. Sessions on the failed leader are still lost; the gain is that clients find the new leader automatically. Use active/active discovery for ordinary batch HS2 and active/passive only where your distribution documents it for interactive query; check its documentation for the namespace and driver settings, which differ between versions.

Worked example: sizing a three-node pool

Take a warehouse with 400 BI users at peak, of whom about 60 have queries running at any moment, plus 30 scheduled ETL sessions. Size the pool so that losing any one node still leaves enough capacity: N+1.

• Sessions per node. Each open session holds a Thrift worker thread in binary mode. The default hive.server2.thrift.max.worker.threads is 500. With 430 sessions over three nodes, each carries about 145, and after losing one node about 215, comfortably under the cap.
• Heap. HS2 memory grows with concurrent compilations, large query plans and result sets buffered for fetch. Starting at 16 to 24 GB per node is common for this load; watch heap after full collections and raise it before pauses grow, not after. Hive monitoring lists the metrics to watch.
• Tez sessions. With hive.server2.tez.default.queues and hive.server2.tez.sessions.per.default.queue, each HS2 keeps a pool of Tez application masters warm. Pools are per server, so three nodes with four sessions on two queues hold 24 AMs in YARN. Budget that container memory; it is the hidden cost of more HS2 nodes.
• Timeouts. BI tools leave idle sessions open for hours. Set hive.server2.idle.session.timeout and hive.server2.idle.operation.timeout so abandoned sessions and unfetched results are reclaimed, and hive.server2.session.check.interval so the check runs. Without them, sessions accumulate on long-lived nodes and the pool becomes unbalanced.

Check the arithmetic against the failure case, not the average. If one node dies at peak, its 145 sessions reconnect to the remaining two within a minute or so, each of which also starts compiling the re-run queries at once. That reconnect storm is the real peak: compile threads, metastore calls and Tez session requests all spike together. Leave headroom for it, or stagger client retries with jitter so they do not arrive in the same second.

Three nodes is the usual minimum: two leaves you at half capacity during any restart. Put them in different racks or zones, and make sure ZooKeeper and the metastore are themselves highly available, or the pool has a single point of failure one layer down.

Rolling restarts without dropping sessions

Restarting an HS2 kills every session on it. To restart without that, remove the node from discovery first and let existing work drain.

# 1. Stop new sessions reaching the old instance (it keeps serving existing ones)
hive --service hiveserver2 --deregister 3.1.3

# 2. Watch it drain: open sessions should fall as clients disconnect or time out
#    (HS2 web UI or metrics; idle-session timeout bounds the wait)

# 3. Once sessions reach zero, or after your maintenance window, stop and restart it

The --deregister option removes registrations for the named version from ZooKeeper while the server keeps running, which was designed for rolling upgrades: start the new version, deregister the old version, and the old server shuts down once its last client disconnects. On distributions managed by Cloudera Manager or Ambari, use their rolling-restart actions, which wrap similar steps. Newer Hive releases also add a graceful-stop mode with a configurable timeout; check whether your version has it before relying on it.

Route every client through discovery rather than a host list. A BI tool hard-coded to hs2-a:10000 defeats every step above. For major upgrades, see the Hive upgrade path.

Failure modes and drills

Rehearse these failures before they happen, by killing nodes in a test cluster and watching what clients do.

• Hard node loss. Its znode expires after the ZooKeeper session timeout, and until then clients may still be sent there and fail. Running queries are lost; Tez containers may linger in YARN until they time out. Clients must retry the connection and the query, so ETL jobs need idempotent steps.
• Zombie server. A node in long GC or with a full thread pool keeps its znode and keeps receiving sessions that hang. Alert on heap and thread saturation, and restart or deregister nodes that cross thresholds.
• ZooKeeper unavailable. New connections through discovery fail even though every HS2 is healthy. Existing sessions continue. Keep a documented direct-host fallback for operators.
• Uneven load. Random selection and long-lived sessions leave one node much busier after restarts. Idle timeouts and connection pools that recycle connections periodically rebalance it.
• Configuration drift. Nodes with different hive-site.xml give different query results or plans depending on which server a client lands on. Deploy configuration from one source and compare effective settings across nodes.
• Metastore bottleneck. More HS2 nodes mean more metastore connections; adding servers can overload the metastore database rather than add capacity.
• Scratch directory leaks. A crashed HS2 leaves query scratch directories behind in HDFS or object storage. Clean them up by age with a scheduled job, or they slowly fill quotas.

Trade-offs

ZooKeeper discovery is built in, needs no extra infrastructure and works with Kerberos without principal tricks, but its load spreading is random, it cannot detect a sick but registered server, and every client needs ZooKeeper access. A load balancer gives one stable endpoint, real health checks and smarter balancing, at the cost of stickiness configuration, Kerberos principal work and another component to operate. Active/passive adds automatic leader election for interactive coordinators but leaves standby capacity idle. More nodes add capacity and shrink each failure's blast radius but multiply Tez pool cost and metastore connections. In every option, a lost node loses its sessions; resilience comes from client retries, not from HS2.

What to do next

1. List what your users keep in HS2 sessions (temp tables, SET variables, long fetches) and make sure clients can reconnect and re-run.
2. Enable dynamic service discovery on every node with the same namespace, and move every client to a ZooKeeper-based or load-balancer URL.
3. Run at least three nodes across failure domains, sized so N-1 nodes carry peak sessions under the worker-thread cap.
4. Set idle session and operation timeouts, and budget the YARN memory for each node's Tez session pool.
5. Alert on heap after GC, thread saturation and open sessions per node, and remove zombie nodes from discovery.
6. Script rolling restarts with deregistration and draining, and test them on a staging cluster.
7. Rehearse node loss, ZooKeeper loss and metastore saturation, and record what each client tool does.