Prometheus 3.16 Release Candidate: Query Cost Limits, Subquery Sample Statistics and What to Test

What the release candidate adds

Prometheus 3.16.0-rc.2 was published on 28 October 2026. It is a release candidate, so treat it as something to test, not to deploy. Its notes list many changes; the one with the most operational weight is experimental query cost estimation, actual cost reporting and configurable execution limits, enabled with --enable-feature=query-cost, from pull request 20028. The same release changes the query statistics so that subquery samples are included in TotalSamples, and fixes several TSDB bugs, including a loss of out-of-order samples when a WAL checkpoint dropped a series record that the out-of-order log still needed.

Three separate things behind one flag

The documentation is explicit that the estimate is never used to reject a query. Limits are checked while the query runs, against what it actually loads and reads, and apply to every query the engine evaluates, including recording and alerting rules. A rule that exceeds a limit fails to evaluate like any other query error, and with the flag on, query_max_duration replaces --query.timeout for rule queries. That last point is the one to plan for: turning on a samples limit for dashboards can quietly break an expensive recording rule.

A local run of the release candidate

For this guide the Windows build of 3.16.0-rc.2 was downloaded, its SHA-256 checked against the release's checksum file, and run on one machine with the flag on, scraping only itself every second, with query_max_series: 500 and query_max_samples_read: 200000. The figures below are from that small instance, a few minutes old, and say nothing about performance at scale.

Two observations. The series estimates matched the actual counts exactly in these runs, while sample estimates were high, by 13 percent for the instant query and 76 percent for the range query; part of the range window preceded the server's first samples, which may explain some of the gap, and the documentation warns that estimates can be wrong in both directions. And the rejection counter worked as documented: prometheus_engine_query_rejected_total showed 4 rejections with reason="max_series" and 1 with reason="max_samples_read" after the probe.

Configuration and the fields you get back

Illustrative values; choose limits from your own cost=true measurements.

# prometheus.yml (with --enable-feature=query-cost on the command line)
global:
  scrape_interval: 15s
  query_max_series: 50000          # reject a query once it loads more series
  query_max_samples_read: 50000000 # reject once it reads more samples
  query_max_duration: 2m           # replaces --query.timeout, rule queries included

The limits are reloadable with the rest of the configuration, and they have no effect unless the flag is set. In a cost=true response, seriesTouched sums series per selector, so a query that uses the same selector twice counts its series twice; samplesRead counts floats as one unit and weights native histograms by their in-memory size; and peakSamples, the peak held in memory, appears only in the actual figures because there is no estimate for it. The release also changes statistics accounting: subquery samples are now included in TotalSamples, so dashboards that track query sample totals will show a step up for subquery-heavy queries after the upgrade, without any change in real work.

Per-request ceilings for callers you trust less

A client can tighten, never loosen, the limits for one request with max_series, max_samples_read and max_query_duration. That is useful for any automated caller that composes queries, from a dashboard builder to an agent tool that writes PromQL. Give such a tool a small budget per query, distinct from the server ceiling, and handle a cost_limit error by narrowing the query instead of retrying it.

Illustrative python, checked for syntax; adapt names and storage to your system.

import json, urllib.parse, urllib.request, urllib.error

def run_promql(base, query, max_series=200, max_samples=50_000):
    """Run an instant query with a per-request budget; raise on cost_limit so the caller narrows it."""
    params = urllib.parse.urlencode({'query': query, 'max_series': max_series,
                                     'max_samples_read': max_samples})
    try:
        with urllib.request.urlopen(base + '/api/v1/query?' + params, timeout=30) as r:
            return json.loads(r.read())['data']['result']
    except urllib.error.HTTPError as e:
        body = json.loads(e.read() or b'{}')
        if e.code == 422 and body.get('errorType') == 'cost_limit':
            raise ValueError('query too broad: add label filters or shorten the range') from None
        raise

The estimate endpoint can serve as a pre-check for a preview ("this query will touch about 800 series"), but because it is advisory and occupies a query concurrency slot, the enforced limits remain the actual guard.

What to test before 3.16 is final

1. Run your recording and alerting rules against a staging Prometheus with the flag on and the limits you intend; count rule evaluation failures. Rules are the likeliest casualty.
2. Compare cost=true estimates with actuals for your heaviest dashboards to learn how far off the estimate is for your data.
3. Decide the response a client should get: a cost_limit error is HTTP 422, while hitting the duration limit usually appears as an ordinary timeout with HTTP 503.
4. Graph prometheus_engine_query_rejected_total by reason and alert on sudden increases after rollout.
5. Check that your query statistics consumers expect the new subquery accounting in TotalSamples, since totals for subquery-heavy queries will rise.
6. If clients reach Prometheus through a proxy or a dashboard data source, confirm that the new max_series, max_samples_read and cost parameters pass through unchanged, or set them at the proxy for every client it serves.

Limits of the feature

• It is experimental and behind a flag; names and behaviour may change before or after 3.16.0.
• Estimation is unavailable in agent mode, and with remote read configured it falls back to assuming the global scrape interval.
• The estimate occupies one of the --query.max-concurrency slots while it runs; a pre-check on every request adds load.
• Limits bound one query, not a client that sends many; rate limiting per client is still a separate job.

Practice and related reading

Further reading:

• Prometheus architecture
• Prometheus at scale
• Metric cardinality management

Sources and scope

Checked against these sources on 1 November 2026. Event: Prometheus v3.16.0-rc.2 release candidate, 28 October 2026; recheck it before acting. The local results were measured on 1 November 2026 with the checksum-verified Windows build of 3.16.0-rc.2 on a single self-scraping instance; they demonstrate behaviour, not performance. Other statements are from the release notes and documentation.

• Prometheus v3.16.0-rc.2 release notes (28 October 2026)
• Prometheus v3.16.0-rc.2 documentation: feature flags, query cost
• Prometheus v3.16.0-rc.2 documentation: HTTP API, query cost endpoints
• prometheus/prometheus pull request 20028: Add Query Cost API