HBase clusters fail predictably, and by the time an alert reaches an on-call engineer, the cluster is usually already recovering or already gone. The goal of alerting is not to react, but to anticipate. This article covers which metrics to instrument, where to set thresholds so you catch problems before they cascade, how to integrate alerts into your on-call workflow via Prometheus Alertmanager, PagerDuty and Slack, and how to tune alert rules so you hear about real problems and not the noise that burns out on-call engineers.
Why alert on HBase at all
HBase problems rarely announce themselves with a single catastrophic failure. Instead, they develop as a series of slow degradations: read latency creeps up over hours, compaction backlog grows in the background, garbage collection pauses lengthen, and region recovery stalls. If you wait for client timeouts or application errors, you are already past the window where a quick tweak (compaction policy, cache flush, load rebalance) would have prevented an outage. Smart alerting gives you 30 minutes of warning before the problem becomes acute.
The other reason to alert on HBase specifically is that its problems are often invisible to application-level monitors. Your web server latency may look fine while HBase is warming up its block cache or recovering a region. Your memory footprint may stay constant while compaction backlog is silently accumulating in HDFS. By the time your application layer sees a problem, the root cause is already several layers down in HBase. Alerting on HBase metrics before your application feels the pain is the only way to stay ahead.
The four families of HBase metrics worth alerting on
Not every metric needs an alert. Instrument everything, but alert only on the few metrics whose abnormal state you can actually act on and whose early rise predicts user-facing problems. HBase exports thousands of metrics via JMX; focus on these four families.
1. Read and write latency. Track the 50th, 95th, and 99th percentile latency for Get and Mutate operations, measured at both the client and the RegionServer. A rise in Get p99 latency often precedes compaction problems or cache misses; a rise in Mutate latency often signals write-path contention or WAL flushing overhead. Alert when p99 latency exceeds your SLO for 5 minutes or when it rises more than 50% over a rolling baseline. The key metric names in HBase metrics2: hbase.regionserver.Get_num_ops, hbase.regionserver.Mutate_num_ops, hbase.regionserver.Get (for latency histogram), hbase.regionserver.Mutate.
2. Compaction backlog and queue depth. When compaction cannot keep up with writes, HBase accumulates a queue of HFiles waiting to be merged. This backlog signals that the cluster is under sustained write load and will soon start throttling writes or delaying flushes. Alert when hbase.regionserver.compactionQueueLength exceeds 50 for more than 10 minutes or when the number of store files (hbase.regionserver.storeFileCount) rises above a baseline threshold. Do not wait for the backlog to hit 1000; by then, user writes are already stalled.
3. Regions in Transition (RIT). When a region is stuck recovering from a failed RegionServer or waiting for the HMaster to reassign it, it is in transition. Clients get stuck trying to reach that region. Alert immediately on any RIT lasting more than 60 seconds; if a single region is still recovering after a minute, something is deadlocked in the HMaster or ZooKeeper. The metric hbase.master.ritCount should be 0 or very small in a stable cluster; alert at > 5 or > 1% of total regions.
4. Garbage collection pauses. HBase RegionServers are JVM processes and pause during garbage collection. Long GC pauses cause regions to be marked as dead and trigger unnecessary failovers. Alert on jvm.gc.pause when the p99 pause exceeds 500ms or when the total GC time in a minute exceeds 10 seconds (25% of wall-clock time). Also alert on the heap usage ratio: alert when heap usage exceeds 85% for more than 5 minutes, because GC pauses grow exponentially as heap fills.
Setting thresholds without alert fatigue
The hardest part of alerting is not choosing which metrics to track; it is choosing thresholds that fire when things are actually wrong, not when the metric merely fluctuates. A threshold set too low creates alert fatigue. You get woken up at 3am for a p99 latency that resolved itself in 30 seconds; after a week of this, you stop believing alerts and miss the one that is real.
Start with these rules: (1) Alert on anomalies, not absolute values. Instead of alerting when latency exceeds 100ms, alert when latency rises 50% over the last 10 minutes or when it is more than 2 standard deviations above the rolling average for the past hour. Prometheus's predict_linear and avg_over_time functions make this easy. (2) Set thresholds from production baseline, not from theory. If your cluster normally sees p99 Get latency of 20ms, set your alert at 40ms or higher. If it normally sees compaction backlog of 10-30 HFiles, alert at > 100. Measure your own baseline for a week before shipping alerts. (3) Require duration thresholds, not single-sample breaches. Alert only after a condition persists for at least 5 minutes. This filters out transient spikes caused by scheduled maintenance, bulk loads, or client retries.
Example Prometheus rule for Get latency:
- alert: HBaseGetLatencyHigh
expr: |
rate(hbase_regionserver_Get_sum[5m]) /
rate(hbase_regionserver_Get_count[5m]) > 50
for: 5m
labels:
severity: warning
annotations:
summary: 'Get latency > 50ms on {{ $labels.instance }}'
runbook_url: 'https://internal.company.com/runbooks/hbase_latency.html'
Notice the for: 5m clause; without it, you fire on every spike. Also notice the runbook URL; every alert should link to a playbook that explains what to do. That link is what makes an alert actionable at 3am.
Prometheus Alertmanager + PagerDuty + Slack integration
Prometheus fires alerts when rules breach; Alertmanager routes them to your on-call system. A typical setup has Alertmanager push to PagerDuty (for on-call escalation) and to a Slack channel (for visibility), with different routes for warning vs critical severity.
Alertmanager configuration sketch:
route:
receiver: 'default'
group_wait: 10s
group_interval: 10s
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: pagerduty-critical
- match:
severity: warning
receiver: slack-warnings
receivers:
- name: pagerduty-critical
pagerduty_configs:
- service_key: '{{ env "PD_INTEGRATION_KEY" }}'
description: '{{ .GroupLabels.alertname }} on {{ .GroupLabels.job }}'
- name: slack-warnings
slack_configs:
- api_url: '{{ env "SLACK_WEBHOOK_URL" }}'
channel: '#hbase-alerts'
Key tuning knobs: group_wait batches related alerts before sending, reducing noise; repeat_interval re-fires resolved alerts after 4 hours to prevent silent failures; routes let you send critical alerts to PagerDuty (which escalates if you do not respond) and warnings to Slack (where you can see them but nobody gets woken up). Adjust these based on your on-call policy and team size.
Trade-offs: over-alerting vs under-alerting
Under-alerting is a catastrophic failure mode: if your alerts are silent, you miss problems and learn about outages from customers. Over-alerting is a slow burn: on-call engineers stop trusting alerts, stop responding quickly, and eventually ignore the on-call channel entirely.
Most teams overweight the cost of over-alerting when tuning thresholds. They set thresholds so high that they only fire when the cluster is already in crisis. This is wrong. A better strategy: start with aggressive thresholds that are probably too sensitive. Monitor for one week. For every alert, ask two questions: (1) Was this actionable? Could an on-call engineer have done anything useful? (2) Did this predict a real problem? Or was it a transient spike that resolved on its own? Disable alerts that answer 'no' to both questions. Do this iteratively and you will converge on a set of alerts that fire rarely but never miss.
One more practice: maintain a runbook for every alert. Every Prometheus alert annotation should link to a document that explains: (1) what the metric means, (2) what caused this alert in the past, (3) the first three things to check, and (4) when to escalate to the database team vs the infrastructure team. Runbooks reduce mean-time-to-recovery (MTTR) by 50% because the on-call engineer does not have to guess what to do at 3am.
Specific alert rules for HBase clusters
Here are the five alerts every HBase cluster should have:
| Alert | Condition | Threshold | Runbook Pointer |
|---|---|---|---|
| RIT Stuck | Regions in Transition > 1 minute | ritCount > 5 for 60s | Check HMaster logs for stuck region assignment; check ZooKeeper for split-brain; restart HMaster if needed |
| Get Latency High | p99 Get > 2x baseline | baseline + 50ms or 50% rise for 5m | Check compaction backlog; check block cache hit rate; check for full GC pauses |
| Mutation Latency High | p99 Mutate > 2x baseline | baseline + 50ms for 5m | Check WAL flushing; check memstore size; check if a RegionServer is running GC; check write rate vs capacity |
| Compaction Backlog | HFiles accumulating | compactionQueueLength > 50 for 10m OR storeFileCount > 15 per store | Increase compaction concurrency; increase major compaction frequency; check if a region is hotspotted |
| GC Pause Long | JVM stop-the-world pause | gc.pause p99 > 500ms | Heap usage > 85%; check if RegionServers have enough heap; tune GC settings; restart RegionServers if pauses persist |
Customize thresholds for your cluster by measuring baselines in production for at least one week. Store these thresholds in version control alongside your Prometheus rules so they stay in sync and you can trace when and why they changed.
Observability beyond alerting: metrics, logs, traces
Alerting is only one layer of observability. When an alert fires, you need to understand not just that the problem exists, but why. This requires: (1) Metrics dashboards showing latency, throughput, and resource utilization over time so you can see the shape of the degradation; (2) Log aggregation from HMaster and RegionServers so you can search for errors and warnings around the time the alert fired; (3) Distributed traces showing the path a single request took through HBase so you can see where it got stuck. Most teams invest only in alerting; teams with low MTTR invest equally in all three.
For HBase metrics, use a Prometheus scrape of the JMX endpoint or deploy Telegraf to forward metrics. For logs, ship HMaster and RegionServer logs to your log aggregation platform (ELK, Datadog, Splunk). For traces, instrument the HBase client library to emit span events on every RPC; this is expensive but worth it for critical paths. The combination of metric + log + trace will let you solve 90% of production problems without ever logging into a machine.
Production checklist
Before you declare alerting done: (1) Every alert has a runbook with specific troubleshooting steps. (2) Thresholds are based on one week of production baseline, not guesses. (3) Alerts are routed to the right person (on-call HBase engineer or DBA team) via PagerDuty or equivalent. (4) Low-severity alerts go to Slack only, high-severity to PagerDuty. (5) Alertmanager de-duplication and grouping are tuned to avoid alert storms during cascading failures. (6) You have tested an alert firing by actually triggering the condition; do not discover at 3am that your PagerDuty integration is broken. (7) Every RegionServer and HMaster is exporting JMX metrics and Prometheus is scraping them reliably. (8) You have a weekly review meeting where you look at all fired alerts from the past week and ask 'should this threshold change?' Alerting is not a one-time setup; it is a continuous feedback loop that improves with practice.