Consumer lag: processed versus committed

Separate processing progress from committed progress

Consumer lag depends on which position you measure. This experiment has one ordered, unbounded log and one consumer. Log end counts emitted records; processed offset is the next position after completed records; committed offset checkpoints that progress periodically. Processing lag is log end minus processed offset. Committed lag is log end minus committed offset. Their difference is work processed but not yet checkpointed, called replay exposure here. A nonzero lag is never relabeled caught up merely because it is small.

Set arrival and service rates

Producer rate is offered records per modeled second. Initial consumer capacity applies before Recovery starts at; Recovery capacity applies from that boundary onward. Set all three rates independently between zero and two hundred. Recovery starts at accepts zero through sixty seconds: zero uses recovery capacity from the beginning, while sixty keeps the initial rate throughout this run. Changing an input rebuilds the same deterministic sixty-second experiment and pauses at its start. Rates describe available processing capacity; an empty queue cannot consume unused capacity or save it for later.

Choose the checkpoint interval

Commit every selects an interval from one to twenty modeled seconds. A checkpoint stores the current processed offset at each exact interval boundary, after that step’s processing. Between checkpoints the committed line stays flat while processing can continue. This does not slow consumption, and a shorter interval does not create additional processing capacity. The fixture assumes commits succeed instantly and monotonically. There are no failed commits, rebalances, transactional writes or actual restarts. Replay exposure names the records that could be repeated after restoring that checkpoint; it does not claim that replay has occurred.

Create backlog and then catch up

Keep producer at one hundred, initial consumer at fifty, recovery capacity at one hundred fifty, and recovery start at thirty seconds. Inspect thirty seconds: three thousand records have arrived, fifteen hundred have been processed, and processing lag is fifteen hundred. During recovery the consumer has fifty records per second of net surplus, so it drains that backlog over the remaining thirty seconds while new records continue arriving. At sixty seconds the processing lag is zero. Set recovery capacity to one hundred instead: it can keep pace with new arrivals but cannot reduce the existing backlog.

Observe checkpoint lag without slower processing

Keep the default five-second commit interval and inspect twenty-nine point nine seconds. Processed offset is one thousand four hundred ninety-five but committed offset is one thousand two hundred fifty, so replay exposure is two hundred forty-five. At thirty seconds the checkpoint catches up to processed offset fifteen hundred. Increase Commit every to ten and compare the staircase spacing; arrival and processing counts at the same time do not change. Set producer to zero and both capacities to zero to confirm that an empty log stays caught up without inventing records.

Inspect the timeline and actual record identities

Play advances the modeled time over sixty viewing seconds at normal speed. Pause freezes it, Next second advances one second, Restart rewinds the same scenario, and Show final state jumps to sixty seconds. Speed offers half, normal and double pace; Inspect time scrubs forward and backward. Restore defaults resets all scenario inputs. The buffer displays the first twenty-four queued record identifiers with the oldest at top left. The full queued count is never trimmed to fit the picture. Exact evidence reports the current oldest, newest, last consumed, counters and offsets. The graph shows history only up to the inspected time.

Understand the event ordering

The engine advances in one hundred millisecond steps. It admits that step’s arrivals before applying its consumer budget. Integer counts come from differences of cumulative rate budgets, so seven records per second really produces four hundred twenty records over this minute rather than rounding every frame. The recovery phase starts a new service budget. Record ages and counts are modeled values, not measured network latency or wall-clock throughput. Changing playback speed changes inspection pace without changing the computed trace.

Keep the Kafka comparison bounded

This is a single-partition integer-offset teaching model, not a Kafka client or cluster simulator. Record count is not byte backlog, event-time lateness or age of the newest event. The oldest queued record’s age is shown separately. There is no log retention, compaction, offset gap, transaction isolation, partition skew, network delay or consumer group reassignment. A real committed offset is not proof that an external side effect completed exactly once. Use the three displayed positions to state precisely what your own monitoring metric means.