Stream–table duality

Append an event

Build current state from an ordered history

The left-hand changelog retains every event added to this experiment. Applying a non-null value replaces the row for that key; applying a tombstone removes it. The current table is the result of folding the inspected prefix in offset order, not a second independently generated view. The fixture has twelve events involving alice, bob, carol and dave. The model is deterministic, so rewinding to the same offset always reproduces the same rows. No Kafka broker, Flink job or external storage is contacted.

Inspect replacements, deletes and a late event

Use Next event to advance through the prepared records. Alice first becomes online, then cart. At offset five, an event with timestamp eight arrives after events with larger timestamps and replaces alice with offline. This model uses log order, not maximum event time. At offset six, carol receives a tombstone and disappears from the table. At offset seven, bob receives paid again: the stored value stays paid, although its provenance offset changes. At the end only dave remains because later tombstones remove alice and bob. These are replacement records, not additive purchase counters.

Compare the retained log with a compacted prefix

Compacted prefix keeps the last record for each key in the currently inspected prefix and preserves those records’ original offset order. Tombstones are retained in this teaching compaction. The replay table folds that compacted sequence from an empty state. With deletion handling enabled, replay agrees with the table produced by the full prefix. At the end there are twelve full-log events and four compacted records, but only one live row. A table cannot recover the discarded intermediate history: current state and complete history are not interchangeable information.

Reproduce a deletion handling failure

Enable Ignore tombstones, the deliberately incorrect policy. The complete-history table keeps old values for deleted keys. The compacted prefix, however, contains only each deleted key’s tombstone, not the previous value. Ignoring that tombstone when rebuilding from empty cannot restore the stale value. Show final state to see the mismatch between the two tables. Disable the switch to restore the intended replacement-and-delete fold. This failure demonstrates why deletion semantics must be consistent across live processing, compaction and recovery, rather than treating a delete marker as an ordinary event to skip.

Append your own controlled event

Key accepts one to thirty characters. Operation chooses Replace value or Delete key; a delete disables the Value field and emits a null tombstone. Value is text up to fifty characters and can intentionally be empty. Event time is an integer label from zero to nine hundred ninety-nine; it does not schedule execution or reorder the log. Append event adds a record at the next offset and rewinds inspection. Add dave with value offline and an event time of one to demonstrate another older timestamp arriving later. Add a delete for dave to remove the final live row. Empty log clears all records; Load example restores the original twelve and enables normal deletion handling.

Control replay without changing data

Play distributes the complete log across sixty viewing seconds. Pause freezes the inspected prefix. Next event reveals one record, Restart rewinds the same log, and Show final state applies all records immediately. Speed selects half, normal or double viewing pace; Inspect time scrubs backward and forward. A backward scrub recomputes the table from that shorter prefix instead of trying to reverse an update in place. Editing deletion policy rebuilds the comparison and pauses at the start. Exact evidence provides the full selected prefix, live table, compacted records and reconstructed table.

Keep the simplifications explicit

This is one ordered replacement changelog with a small in-memory materialized view. The experiment accepts at most forty records and reports the limit rather than silently discarding earlier events. It does not simulate asynchronous broker log cleaning, tombstone retention expiry, distributed partitions, transactions, joins, aggregations, event-time windows or durable checkpoints. Compaction here is an exact prefix transformation for explanation, not a claim about when a real broker cleans segments. Repeated replacement values do not increment a sum; applying the same event to an external side effect may have different idempotency requirements. All example identities and values are fictional.