Kafka partition routing and expansion

Keys across partitions

Append a record

Route serialized keys into partition-local logs

Each record is appended to exactly one partition and receives that partition’s next local offset. The event number identifies its place in this fictional producer sequence; it is not a Kafka global offset. For non-null keys, this experiment encodes the string as UTF-8, computes Murmur2, clears the sign bit and takes the remainder by the current partition count. That keyed rule follows the referenced Kafka Java producer implementation. The evidence displays serialized bytes, unsigned hash, positive hash, chosen partition and local offset rather than an invented hash label.

Configure a controlled expansion

Initial partitions chooses one through eight starting logs. Expanded partitions chooses the later count and cannot be smaller than the initial count. Expand after records schedules expansion immediately before appending the next record after that many have already been written. Zero expands before the first append; a threshold beyond the available records never triggers. Setting both counts equal disables expansion. Changing a scenario control intentionally rebuilds and rewinds the scenario. Within its replay, expansion adds empty logs and preserves every old record and its offset; it does not redistribute existing data.

See one key span old and new partitions

Load the example, which starts with three partitions and expands to five after ten records. Inspect thirty seconds, when ten of twenty records have been appended: user_42 records are in P1. Next record triggers expansion and the new user_42 record routes to P3. Its older P1 records remain visible. Show final state and inspect the same-key location summary. Changing the partition count changes the modulo calculation for future sends, so you cannot assume a key remains confined to its former partition across this transition. Independent consumers of different partitions do not provide one combined ordering guarantee.

Keep null, empty and literal text distinct

A missing key is represented by a real null, not the text null. The empty string is a valid zero-byte serialized key and is still hashed. The four-character string null is also hashed. Missing-key records use this lab’s deterministic teaching policy: consecutive groups of Null-key batch size records choose successive partitions, starting with P0 and wrapping by the current count. Only missing-key sends advance that counter. This illustrates batching rather than reproducing Kafka’s adaptive, availability-aware, byte-sized sticky policy. It is not a claim that every real producer routes null keys this way.

Append your own key and test a hot partition

Key text accepts up to forty characters, including an empty string. No key disables that field and makes Append record add a null key. Otherwise the exact entered text is appended. Repeatedly add user_42 and show the final state: records after expansion continue to collect in P3 under the unchanged keyed rule. A hot key does not automatically spread across every partition. Try a Unicode key and inspect its byte array to see that hashing operates on encoded bytes, not a language’s generic string hash. Empty scenario removes the log; Load example restores all initial controls and twenty prepared keys.

Use playback to inspect evidence

Play reveals the selected producer sequence over sixty inspection seconds. Pause holds the current prefix, Next record appends one event, Restart rewinds the same scenario, and Show final state reveals the complete log. Speed changes viewing pace between half, normal and double; Inspect time supports backward and forward scrubbing. The partition cards are actual accessible lists, not compressed canvas text. Exact evidence contains all inspected records and per-partition logs. The scenario is limited to forty records and reports the limit instead of discarding history.

Understand the boundaries of this demonstration

One fictional producer appends successfully and sequentially; no broker or account is connected. There are no leaders, replicas, acknowledgements, retries, idempotence settings, transactions, producer races, consumer assignment or throughput measurements. Real partitioning also depends on serializers, explicit partition choices, client configuration and implementation. The simplified null-key batch policy is intentionally separate from the verified keyed hash rule. A hash can place different keys together, and more partitions do not eliminate key skew. This lab demonstrates routing and preserved history, not a production migration or a promise of global ordering.

Keyed rule reference: Apache Kafka Java producer partitioner.