OpenTelemetry Collector 0.163.0: A Sending-Queue Deadlock Under block_on_overflow, Batches That Lost Everything Behind an Oversized Span, and a Hash Change

Recent development · Practical technical guide · 20 October 2026

Three exporter fixes in OpenTelemetry Collector v0.163.0, released 14 October 2026, read from the issues and pull requests: a sending-queue deadlock with block_on_overflow, batch splitting that dropped every item behind an oversized one, and an xhash change to every hash value.

Why a minor collector release deserves a read

OpenTelemetry Collector releases arrive every two weeks or so and most of the changelog concerns component authors. v0.163.0, published on 14 October 2026, contains three bug fixes in the end-user changelog that matter to anyone operating a collector fleet: a deadlock in the exporter sending queue, a data-loss bug when batches are split, and a change to the internal hash used as an identity key. Each was found in practice, and each has a configuration or monitoring implication even after you upgrade. This article goes through them from the issue reports and pull requests rather than the one-line changelog entries.

The deadlock: block_on_overflow and cancelled offers

The exporter helper's sending queue has an option, block_on_overflow, off by default. With it on, a caller that finds the queue full waits for space instead of having its data rejected. Issue #15700 reports what happened in production on v0.157.0: a persistent metrics queue configured with block_on_overflow: true, one consumer and a queue size of 6,000,000 items stopped at 5,995,714 items. The queue and sent counters stopped moving while the health endpoint kept answering.

The mechanism is in the queue's context-aware condition variable. It tracks the number of waiters in a counter and wakes them through a notification channel with room for one signal. When blocked offers are cancelled, the cancelled waiters must re-acquire the queue mutex before they can decrement the waiter count. If two completion signals arrive close together while those cancelled waiters are still queued for the mutex, the first signal fills the channel and the second blocks on the full channel while holding the mutex. Nobody can take the mutex to drain the channel, so reads, offers, completions and even size queries stop. The issue includes a deterministic test that reproduces it without sleeps.

The fix, PR #16169, merged on 13 October, gives the channel one slot per CPU so that a signal rarely finds it full, and in the rare case that it does, releases the mutex while sending and re-acquires it afterwards, so that cancelled waiters can run and free a slot. The issue also documents the workaround for older versions: block_on_overflow: false, at the price that a full queue rejects data instead of waiting.

Splitting batches: one oversized item took the rest with it

When batching in the sending queue has a max_size, the exporter helper splits large requests into batches that fit. PR #15937 describes what went wrong when a single log record, span or data point was itself larger than max_size. The extraction step returned an empty batch, the split function returned early with an error, and the remainder of the request, which still held the oversized item and every valid item queued behind it, was discarded. Items already extracted into earlier batches survived; everything after the oversized one was lost, and the dropped count was wrong.

In practice this needs a size unit in which one item can exceed the limit, which is the bytes sizer: a single span with a very large attribute can be bigger than max_size bytes, while with the items sizer each item counts as one. The fix drops only the oversized item, keeps the rest and reports the real count. It landed for logs in v0.162.0, whose changelog says applied to logs only for now. Issue #16087 tracked the other signals, and v0.163.0 carries the fix for spans and profiles. The metrics port, PR #16091, was still open when this article was checked on 20 October 2026, so metrics pipelines with a bytes-sized max_size remain exposed on 0.163.0.

Illustrative exporter configuration showing the two settings involved.

exporters:
  otlp:
    endpoint: gateway.internal:4317
    sending_queue:
      enabled: true
      queue_size: 5000
      block_on_overflow: false     # the documented workaround on versions before 0.163.0
      batch:
        sizer: bytes
        min_size: 1000000
        max_size: 4000000          # one item larger than this is dropped; before the fix, so was the rest of the request

xhash: every hash value changes

The third fix is small in code and large in consequence. xhash computes hashes of attribute maps that several components use as identity keys for series and groups. PR #16050 explains the bug: map keys, string values and bytes values were written into the hash buffer with a type prefix but no length, so the boundary between fields was ambiguous, distinct maps could encode to the same bytes and therefore the same hash, and a collision silently merged unrelated state. The fix writes a length before every variable-length field.

The PR notes the side effect: all hash values change. Hashes are normally kept only in memory, but anything that derives a stable key from them, and the PR names Kafka partition keys in the contrib repository, will shift once after the upgrade. A follow-up PR documents that xhash values are not stable across versions. If a partition, shard or cache key in your pipeline comes from these hashes, expect one redistribution when you roll out 0.163.0, and roll out in a way that tolerates it.

Worked example: noticing a stalled queue

This example is illustrative. A gateway collector exports metrics through a persistent queue with block_on_overflow enabled, on a version before 0.163.0. At 02:10 the queue size metric stops changing, the sent metric points counter stops increasing, and the health check stays green. Agents upstream begin to back up, then drop data at their own limits. Nothing in the collector's logs explains it.

The signals that would have caught it are the exporter helper's own metrics: otelcol_exporter_queue_size flat while otelcol_exporter_queue_capacity is unchanged, otelcol_exporter_sent_metric_points flat, and no increase in otelcol_exporter_send_failed_metric_points. A stall looks like a queue that is neither growing nor draining and an exporter that is neither succeeding nor failing. Alert on that combination, for example a sent counter that has not increased for ten minutes while the queue is non-empty, because a health endpoint does not see it. After upgrading, keep the alert; it detects any stuck consumer, not only this bug.

Upgrade and configuration checklist

Upgrade to 0.163.0 or later anywhere block_on_overflow is enabled; until then, consider setting it to false and alerting on enqueue failures instead.
Find pipelines that batch with the bytes sizer and a max_size, and check the logs for oversized-item errors; on 0.163.0, metrics pipelines are still affected.
Inventory anything that derives a persistent key from collector hashes and plan for one redistribution.
Add a stall alert: queue size non-zero and unchanged, sent counter flat, failed counter flat.
Read the collector-contrib release that pairs with your core version, since most components you run live there.

Failure modes

Trusting the health endpoint to detect a stalled exporter; it reported healthy during the deadlock.
Turning block_on_overflow on to avoid drops, which converts drops into back-pressure and, on older versions, exposed the deadlock.
Assuming the split fix covers metrics because the changelog mentions spans and profiles.
Huge span or log attributes, such as full request bodies, that make single items larger than a byte-sized batch limit.
A partitioning scheme keyed on xhash values that silently reshuffles at upgrade time.

Sources and scope

Checked against these sources on 20 October 2026. Event: OpenTelemetry Collector v0.163.0 release, 14 October 2026; recheck it before acting. Examples and figures are illustrative, not measurements.

Upgrade where block_on_overflow is on, remember the split fix does not cover metrics yet, plan for xhash keys to change, and alert on a queue that neither grows nor drains.