Hive Monitoring

JMX metrics: the heartbeat of a Hive cluster

HiveServer2, the metastore daemon, and the Tez/LLAP engines all export JMX beans — performance counters for query throughput, operation latency, connection pools, and resource consumption. JMX is the standard way Hadoop components expose internal state to monitoring systems, and Prometheus scrapers listen on a configured port (default 9010 for HS2) and pull those metrics every 15–30 seconds. The beans include compile time, execute time, fetch latency, memory usage, and thread-pool depth.

Why start with JMX instead of logs? Because JMX is instantaneous and always-on. A log line fires after an event; a JMX metric is current when you scrape it. HS2 thread-pool queue depth, current connection count, and active-operation count are live gauges you can graph and alert on; you do not have to wait for a query to finish logging to know the system is under stress. When dashboard load spikes at 8am and the team wonders if Hive is the bottleneck, JMX metrics answer in seconds. Logs tell you why after the fact; metrics tell you during.

The catch: JMX beans are cumulative and ephemeral. Histogram metrics like total-queries-run and total-bytes-returned are counters that only go up; Prometheus deals with that through rate() functions. But connection count and active-operations are gauges: when an HS2 instance crashes, those gauges vanish, and a naive alert on 'active operations > 0' becomes useless during a rolling restart. Good monitoring accounts for instance churn — alerting on the fleet average or the per-instance percentile rather than an instance's moment-in-time state.

Setting up Prometheus scraping and JMX exporters

HS2 ships with a built-in JMX remote interface. Start it by setting Java options in your startup script or via HIVE_CONF_DIR:

export SERVER_JVMARGS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Then plug a Prometheus JMX exporter in front. The standard approach is a sidecar container or a standalone exporter on each HS2 host that connects to the JMX port and serves Prometheus-format output:

scrape_configs:
  - job_name: 'hiveserver2'
    static_configs:
      - targets: ['localhost:9010']
        labels:
          instance: 'hs2-node-1'
          cluster: 'prod-warehouse'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 'jmx-exporter:5556'  # exporter host

The JMX exporter translates bean names to Prometheus metric names, usually with underscores and a prefix. org.apache.hadoop.hive.hs2.metrics.JmxMetrics.CompileTime becomes hive_hs2_compile_time_ms. Once scraping runs, your time-series database (Prometheus, Grafana Loki, or InfluxDB) indexes every metric; from there, graphing and alerting are standard.

Critical metrics to track for HS2 and metastore

Not all JMX metrics matter equally in a running cluster. The top tier to alert on tells you when capacity is exhausted or work is blocked:

The second tier is diagnostic — you drill into these when something is already wrong:

• hs2_parse_time_ms: time to tokenize and parse SQL. High values mean unusually complex queries or a meta store already slow.
• hs2_fetch_time_ms: time to assemble and send result rows. A spike suggests result spooling kicked in or the client is slow to read.
• metastore_direct_sql_calls: count of direct SQL reads from HMS backing store. High count = metadata cache miss rate is high.
• ranger_audit_events_latency: if Ranger is slow, authorization is the compile bottleneck.

Query logging: from HS2 to HDFS or Elasticsearch

JMX metrics are aggregates; to diagnose why one specific query was slow or wrong, you need the query log. Hive can write per-query records to a spool directory, emit them to a log aggregator, or both. The log includes the SQL text, compile time, execute time, user, queue name, and eventual result (success, timeout, error).

Option 1: Query logs to HDFS. Hive flushes query records to a directory like /logs/querylog/HS2_INSTANCE_1/ as delimited text or Parquet. Pros: no external system needed, data is queryable with Hive itself, cheap at massive scale. Cons: you have to periodically compact and archive logs, or they outlive their usefulness on disk.

hive.server2.logging.operation.enabled=true
hive.server2.logging.operation.level=PERFORMANCE
hive.server2.logging.operation.log.location=/hive/logs/hs2-operations
hive.server2.operation.log.query.result.format=json

Option 2: Logs to Elasticsearch. A log shipper (Filebeat, Logstash) tails the local HS2 logs or reads from a query-result API and pushes records to Elasticsearch. Pros: real-time search and aggregation, UI dashboards out of the box, long-term retention with automated rollover. Cons: another cluster to operate, network I/O to remote sink.

In production, most teams use both: lightweight JSON query logs to HDFS for long-term record, and (sampled or all) logs to Elasticsearch for the hot queries of the last 7 days. The query log record typically includes:

• statement ID, user, queue, timestamp of execution
• SQL text (truncated if huge)
• Compile time, execute time, fetch time in ms
• Bytes read/written and task count
• Exception or error code if not successful

Diagnosing slow queries — the compile vs. execute split

When someone reports this query is slow, the first split is: did the metastore or authorization system choke (compile phase), or did Tez/LLAP choke (execute phase)?

Check your query log for compile_time_ms vs. execute_time_ms. If compile is >5 seconds and execute is normal, something in metadata or Ranger is saturated. Tools to drill down: query the metastore JDO connection pool, check if the query touched a huge table with millions of partitions (metastore cache miss = sequential disk reads), or check Ranger audit logs for long authorization round-trips.

If execute time is high, the problem is in Tez/LLAP — data is moving slowly, shuffle is bottlenecking, the cluster is full, or the plan itself is bad (wrong join order, missing statistics). Use the Tez UI (if enabled) or tez_task_attempt_latency metrics to see which stages are running long.

A practical pattern: add a compile_time alert (p95 > 5s) and add an execute_time alert (p95 > 30s). When compile spikes, page the platform team to check metastore and Ranger. When execute spikes, check cluster saturation and ask the user if the query is new or changed.

Metastore performance and bottleneck detection

The Hive metastore daemon is a JDBC client talking to a backing SQL database (MySQL, Postgres, etc.). The connection pool has a size limit (default 10); when all connections are in use and a new metadata request arrives, it queues. If the queue backs up, compile latency rises and queries pile up.

Watch these metrics:

• metastore_jdo_connections_in_use — should stay <80% of pool_size. If it climbs to 100%, new metadata reads block.
• metastore_direct_sql_latency_ms — time to fetch a partition or table from the backing DB. Spikes = database under load or query is fetching too many partitions at once.
• metastore_objects_in_cache — number of tables and partitions in the in-memory cache. A stable number means cache hit rate is working; a sudden spike means a query forced a massive cache population (sign of a query on a huge table with no partition pruning).

When the metastore backs up in production, the fix is usually short-term and long-term:

1. Short-term: kill long-running compile phases (via HS2 operation cancel), which frees metadata connections for new queries.
2. Long-term: scale the metastore connection pool, add more replicas if your HMS daemon is single-threaded, or shard metadata across multiple HMS instances by table namespace.

Alerting on HS2 saturation and queue backlog

HiveServer2 has internal thread pools for parsing, compilation, and execution submission. When a pool is full, new work queues. A queue depth that climbs and doesn't drain is the early warning sign of overload.

Define alerts in your Prometheus config (or Grafana):

- alert: HSQueue_Backlog
  expr: hive_hs2_active_operations > 100
  for: 5m
  annotations:
    summary: "HS2 queue backlog on {{ $labels.instance }}"
    description: "{{ $value }} operations queued; cluster overloaded."

- alert: HS2_ConnPool_Saturation
  expr: (rate(hive_hs2_active_connections[1m])
         > bool 0.9 * hive_hs2_max_connections)
  for: 2m
  annotations:
    summary: "HS2 connection pool at >90% on {{ $labels.instance }}"
    description: "May reject new connections."

- alert: HS2_CompileTime_High
  expr: histogram_quantile(0.95,
         hive_hs2_compile_time_ms_bucket) > 5000
  for: 10m
  annotations:
    summary: "HS2 compile p95 > 5s on {{ $labels.instance }}"
    description: "Metastore or Ranger is slow; check connection pool."

Pair each alert with a runbook: what metric to check next, which log to tail, and which command to run to reduce load (kill a user's job, roll restarts, etc.). Alerts without runbooks are noise.

LLAP and Tez execution performance metrics

When a query runs against LLAP (the in-memory cache layer) or Tez (the DAG execution engine), the execution phase is instrumented at the task level. Tez publishes counters for bytes shuffled, spilled to disk, output record count, and wall-clock latency per task.

Key Tez metrics:

• tez_dag_completion_time — how long from DAG submit to finish. Baseline this per query type; a spike suggests cluster congestion.
• tez_shuffle_bytes — how much data moved in the shuffle phase. If this is huge and execution is slow, shuffle bandwidth is the bottleneck (add faster network or reduce data shuffled via optimization).
• tez_spilled_bytes — data written to disk during shuffle (means reducer buffer overflowed, increase reducer memory).
• tez_task_attempt_latency_ms — per-task wall clock. If the p99 is much higher than median, you have straggler tasks (a subset of reducers running long). Common causes: uneven join key cardinality, a reducer hitting a much-larger input than its siblings, or scheduler picking slow nodes.

For LLAP monitoring, track daemon health and cache hit rate:

• llap_cache_hit_rate — percentage of reads satisfied from the in-memory cache. If this drops, either the cache is undersized or the query pattern shifted (e.g., new table, wider columns).
• llap_eviction_count — times data was evicted to make room. A rising trend means you need a larger cache or need to pin hot tables.
• llap_task_queue_depth — tasks waiting for a free executor. If this is consistently high, increase LLAP task parallelism.

Building dashboards and setting up long-term storage

Prometheus retains metrics for 15 days by default. For long-term trend analysis (capacity planning, cost tracking), push metrics to a scalable time-series store (InfluxDB, Thanos, Cortex) or a data warehouse. A daily Hive query can roll up hourly averages and percentiles from HDFS query logs into a summary table, which you then graph in Grafana or Looker.

A minimal dashboard includes:

• HS2 health: active connections, active operations, thread pool queue depth — shows whether the service is under load.
• Latency distribution: compile time p50/p95/p99, execute time p50/p95/p99 — shows if you're meeting SLAs and whether tail latency is worsening.
• Throughput: queries per minute, bytes read/written per hour — shows cluster utilization trend.
• Error rate: failed queries, timeouts, OOM errors — early sign of misconfigs or resource exhaustion.
• Metastore health: connection pool usage, HMS response time — show whether metadata is the bottleneck.
• LLAP / Tez task latency: task attempt p95, shuffle bytes, spilled bytes — show execution-phase efficiency.

Make the dashboard filterable by queue, by user, by date range, and by execution engine (Tez vs. LLAP) so teams can slice the data for their own workload without seeing everyone else's.

Common monitoring pitfalls and how to avoid them

Pitfall 1: Alerting on raw gauge values on ephemeral instances. If HS2 is a fleet in ZooKeeper discovery mode, an instance may come and go; its metrics vanish when it crashes. An alert like 'active_connections > 0' is true on running instances and spuriously fires when instances restart. Fix: Alert on the fleet average or per-instance percentile over a time window, or require the alert to be true for several scrape cycles, so one instance restart does not trigger.

Pitfall 2: Missing the difference between compile and execute slowdown. Slow-query reports often lump both phases together. If you do not split them in your alerts and dashboards, you waste time checking the wrong system. Fix: Log and graph compile_time_ms and execute_time_ms separately, and have separate runbooks for each (metastore vs. cluster saturation).

Pitfall 3: Ignoring query log volume. Query logs grow fast at scale (1 query per second on a busy warehouse = 86M rows/day). If logs land in Elasticsearch without retention policy, costs balloon; if they land in HDFS and are never pruned, they consume space. Fix: Set a retention window (e.g., 7 days hot in Elasticsearch, then aged to HDFS; 30 days in HDFS, then archived to S3). And sample logs if possible — logging 100% of short queries is usually waste; 1% sample is enough for latency percentiles.

Pitfall 4: Not tracking per-user or per-queue metrics. A cluster-wide average can hide one team consuming all resources. Fix: Tag every metric with queue, user, and execution engine. This costs cardinality but answers the real on-call question: 'Whose job is making the cluster slow?'

Triage playbook for production incidents

When HS2 latency spikes and the on-call page fires, follow this sequence:

1. Check JMX gauges first (15 seconds): HS2 active operations, metastore connection pool in-use, Tez task latency. Is the cluster actually under load, or is this a false alarm?
2. Check fleet health (30 seconds): How many HS2 instances are up? Any recent restarts or crashes? If an instance just died, congestion might be traffic re-routing to the remaining instances (temporary).
3. Compute the compile/execute split (1 min): Look at the last 100 queries in the query log (Elasticsearch) or last hour from HDFS logs. Are they slow at compile or execute? Slow at both? If compile is the outlier, check metastore. If execute, check Tez/LLAP.
4. Metastore deep-dive (2 min, if compile was slow): Query show processlist on the metastore backing database (or use HMS debug commands). Is a query holding locks? Fetch the statement that's taking longest and see if it's a massive partition scan. If so, recommend the user add a partition filter.
5. Tez/LLAP deep-dive (2 min, if execute was slow): Check the Tez UI for stuck stages or task failures. Check LLAP daemon logs for GC pauses or cache evictions. Use YARN resource manager to see if the cluster is at capacity (all CPU or memory allocated).
6. User communication (1 min): Identify the worst queries (by duration) and their owners. Page the team: 'Query X ran 5 min slower than usual; cluster load is high; consider waiting an hour or optimizing the query.'
7. Prevention (post-incident): If this was a new query, ask the user to provide statistics or consider automatic query optimization. If metastore was the bottleneck, increase pool size or add HMS replicas. If Tez/LLAP was full, scale the cluster or implement workload management (limits per user/queue).