Kafka Partitioning Strategies

A Kafka topic is a set of partitions, and a partition is the only place Kafka keeps order. The partition key a producer attaches to each record therefore decides far more than load balancing. It decides which records stay in sequence, how many consumers can work in parallel, where a stream processor keeps the state for an entity, and which records a compacted topic treats as versions of the same thing. Pick a good key and the rest of the pipeline is straightforward. Pick a bad one and you get hot partitions, events applied out of order, joins that quietly miss data, and a painful migration the first time the topic has to grow.

This article treats partitioning as a design decision rather than a producer setting. It covers what the key controls, a framework for choosing it, how to size the partition count, why joins need co-partitioning, how consumer assignment interacts with the key, and a worked migration that grows an account ledger from 12 to 48 partitions without breaking per-account order. The mechanics of the producer partitioner itself are only summarised here, because Kafka partitions from the client side covers them in detail.

The key picks the partition, the partition picks the consumer, and a co-partitioned table keeps joins local, until the partition count changes.

What the partition key controls

Four properties follow from the key. Order: Kafka guarantees that records in one partition are read in the order they were appended, and makes no promise across partitions. Records that must be applied in sequence, such as the debits and credits of one account, need to share a partition, which in practice means they need to share a key. Parallelism: in a consumer group each partition is read by exactly one member at a time, so the partition count is the ceiling on consumer parallelism. State locality: a stream processor that aggregates per key keeps that state next to the partition it reads, so the key is also the sharding key for your state stores. Compaction identity: on a compacted topic the broker keeps the latest record per key within each partition, so the key is the identity of the thing being versioned.

The partitioner turns a key into a partition number. In the Java client a record with a non-null key goes to toPositive(murmur2(keyBytes)) % numPartitions. A record with a null key is spread by the built-in uniform sticky logic introduced by KIP-794 in Kafka 3.3: the producer fills a batch for one partition, then moves on, and steers away from brokers that are slow to accept data. The older DefaultPartitioner and UniformStickyPartitioner classes were deprecated in 3.3 and removed in Kafka 4.0, so the right way to get default behaviour is to leave partitioner.class unset. Two details matter for strategy. Null-keyed records are not round-robin record by record, so do not rely on null keys for exact per-record balance. And other clients do not hash the same way: librdkafka, which underlies the Python, Go and .NET clients, defaults to consistent_random (CRC32). If services written in different languages produce to the same keyed topic, set partitioner=murmur2_random in librdkafka so a key lands where the Java producer would put it.

What the partition key controls

Choosing the key: a decision framework

Start from the question: what is the smallest set of records whose relative order matters? That set is the key. Making the key coarser than necessary, for example keying by tenant when only per-order sequence matters, gives you fewer distinct keys and more skew. Making it finer than necessary, for example keying by event ID when per-account order matters, breaks correctness. The table lists the common choices.

Strategy	Key	Use when	Watch out for
Entity key	`account_id`	State changes for one entity must apply in order	Skew if a few entities dominate traffic
Composite key	`tenant_id:order_id`	Order matters per sub-entity; tenant needed for routing or ACLs	Every producer must build the string identically, byte for byte
Bucketed key	`sensor_id:hour`	Order matters only within a time window	Order across windows is lost
Salted key	`account_id#0..7`	One huge entity needs several consumers	Per-entity order is gone; consumers must merge or tolerate it
Null key	none	Telemetry and logs where only aggregates matter	No ordering between any two records
Explicit partition	producer sets the partition	Rare, such as reserving partitions for priority traffic	Couples producers to N; breaks silently when N changes

Two rules apply. First, the key must be stable for the life of the entity. If an order is keyed by customer_id while unassigned and by account_id afterwards, its events split across partitions at the moment the key changes, and a consumer can see the second half first. Second, measure skew before shipping. Sample a day of production keys, hash them with the producer's real algorithm and count records per partition. If the busiest partition carries more than about twice the median, either choose a finer key or isolate the heavy entities, for instance by giving the top tenants their own topic.

Sizing the partition count

The partition count sets the parallelism ceiling, and for keyed topics it is also part of the key-to-partition mapping. A common sizing rule takes the target throughput T and the measured per-partition throughput of the producer (p) and of the consumer (c), and picks at least max(T / p, T / c) partitions. Measure p and c on your own hardware with your own message sizes and processing logic, because the consumer side is usually the bottleneck when every record triggers a database write.

Then add headroom for growth, because growing a keyed topic later is expensive. Kafka lets you raise a topic's partition count but never lower it, and raising it changes hash % N for most keys. Records already written stay where they are while new records for the same key go to a new partition, so a consumer can process a newer event for an account on one partition before an older event that is still waiting on another. Going from 12 to 48 partitions keeps a key in place only if its hash mod 48 is below 12, which is a quarter of keys; three quarters move. The snippet below computes the same figure for any pair of counts.

import hashlib, random

def part(key: bytes, n: int) -> int:
    # stand-in for murmur2; any uniform hash gives the same fraction
    return int.from_bytes(hashlib.md5(key).digest()[:4], "big") % n

keys = [f"acct-{random.getrandbits(40)}".encode() for _ in range(200_000)]
for old, new in [(12, 48), (12, 13), (12, 24)]:
    stay = sum(part(k, old) == part(k, new) for k in keys) / len(keys)
    print(f"{old}->{new}: {stay:.1%} of keys keep their partition")

Overshooting costs broker file handles, replication work and longer rebalances, so size for two to three years of keyed throughput and plan the migration below for anything beyond that.

Co-partitioning for joins and stateful processing

A stream join reads two topics and matches records with the same key. Kafka Streams and ksqlDB do this without a network shuffle by giving one task partition i of both inputs, which only works if the inputs are co-partitioned: same key, same partition count and the same partitioning algorithm. Kafka Streams checks the partition counts when it starts and fails if they differ. It cannot check that the producers hashed the same way, so a payments topic written by a librdkafka producer with its default partitioner and an accounts table written by a Java producer will join cleanly and return wrong answers, because the two copies of acct-42 sit in different partition numbers.

When the key you need is not the key on the topic, a selectKey or groupBy in Kafka Streams creates an internal repartition topic, writes every record to it under the new key and reads it back. That costs a second write and read of the stream and adds latency, so choose topic keys that match the joins and aggregations you know you will run. See Kafka Streams in practice for how tasks, state stores and repartition topics fit together.

How consumer assignment interacts with the key

The partitioner decides which partition a key goes to. The group's assignor decides which consumer reads that partition. With the classic protocol the Java consumer's default partition.assignment.strategy is the range assignor, with the cooperative sticky assignor also listed so groups can upgrade. Range assignment gives the same consumer partition i of every subscribed topic, which is what a hand-written join consumer needs. Since Kafka 4.0 the KIP-848 consumer protocol is generally available. Consumers opt in with group.protocol=consumer, the broker computes assignments, and group.remote.assignor selects uniform or range; choose range when you rely on co-partitioned input.

Within one partition, a consumer is free to process records concurrently as long as it keeps per-key order, for example by sending each key to a fixed worker thread chosen by hashing the key again. Commit offsets only up to the lowest record that every worker has finished. During a rebalance a partition can move between consumers; consumer rebalancing explains how to avoid reprocessing and out-of-order effects while that happens.

Worked example: growing a ledger from 12 to 48 partitions

A payments service writes debits and credits to ledger.v1, which has 12 partitions and is keyed by account_id. A balance service consumes it, joins against a compacted accounts topic, also 12 partitions, and writes balances. Traffic has tripled and the balance service is pinned at 12 busy consumers. The team wants 48 partitions. Running kafka-topics --alter --partitions 48 on the live topic would move three quarters of accounts to new partitions in the middle of their event streams, and would also break co-partitioning with accounts. Instead they migrate to new topics.

Create ledger.v2 and accounts.v2 with 48 partitions each and the original configuration.
Run a copier that reads each v1 topic from the earliest retained offset and produces every record to v2 with the same key. Use an idempotent producer (the Java default since 3.0; librdkafka needs it set, as below) so retries cannot reorder or duplicate records. Each v1 partition holds all of an account's history in order, and the copier reads it in order, so each account's order survives in v2.
Start a v2 balance service in shadow mode with 48 consumers, writing to a scratch table. Compare its balances with v1 for a sample of accounts until they match.
Cut over the producers. Pause payment writes, wait until the copier's lag on ledger.v1 is zero, point the producers at ledger.v2 and resume; switch the accounts producers the same way. The pause lasts about as long as the copier's lag, usually seconds. Skipping the pause would let a new debit reach v2 before an older debit that the copier had not moved.
Promote the v2 balance service to write the real table, stop v1, and delete the v1 topics after their retention period.

from confluent_kafka import Consumer, Producer

src = Consumer({"bootstrap.servers": BROKERS, "group.id": "ledger-v1-to-v2",
                "auto.offset.reset": "earliest", "enable.auto.commit": False})
dst = Producer({"bootstrap.servers": BROKERS, "enable.idempotence": True,
                "partitioner": "murmur2_random"})   # match the Java producers
src.subscribe(["ledger.v1"])

while True:
    msg = src.poll(1.0)
    if msg is None or msg.error():
        continue
    hdrs = (msg.headers() or []) + [
        ("v1-partition", str(msg.partition()).encode()),
        ("v1-offset", str(msg.offset()).encode()),      # lets consumers dedupe
    ]
    dst.produce("ledger.v2", key=msg.key(), value=msg.value(), headers=hdrs)
    dst.poll(0)
    if msg.offset() % 1000 == 0:
        dst.flush()                    # only commit what v2 has acknowledged
        src.commit(message=msg, asynchronous=False)

The v1-offset header makes the cutover safe to retry. If the copier crashes after producing but before committing, it copies a few records again, and the balance service drops any record whose v1 partition and offset it has already applied. The same pattern works whenever you need to change a topic's key, not only its partition count.

Failure modes and trade-offs

Hot key. One entity, such as a large merchant or a bot account, saturates a partition and its consumer while the others sit idle, and consumer lag grows on that partition alone. Alert on per-partition lag and bytes in, not only topic totals.
Key changes mid-life. An entity re-keyed during its lifetime splits across partitions, so updates can be applied out of order. Derive the key from an immutable identifier.
Mixed client partitioners. Java and librdkafka producers writing to one keyed topic put the same key on different partitions. Per-key order and joins both break, with no error.
Adding partitions to a compacted topic. The latest value for a key may land on a new partition while an older value survives compaction on the old one. A consumer rebuilding state from the beginning can end with the stale value. Migrate compacted topics; never alter them in place.
Explicit partition numbers. Code that writes to "partition 3 for priority traffic" keeps working after a resize but now shares that partition with remapped keys.
Too many partitions. Thousands of small partitions per topic stretch rebalances and producer batching, so records sit longer waiting for a batch to fill and throughput falls.

The trade-off underneath all of these is order against spread. Finer keys spread load and parallelise better; coarser keys preserve more order. Kafka message ordering covers the other places order can break, such as retries, failover and redelivery, once the key is right.

What to do next

Write down, for each keyed topic, the smallest set of records whose order matters, and check that the key matches it.
Hash a day of real keys with your producers' actual partitioner and chart records per partition.
Check that every producer to a keyed topic uses the same partitioner; set partitioner=murmur2_random on librdkafka clients that share topics with Java.
List every join and aggregation, and confirm that its inputs share key, partition count and partitioner.
Size new keyed topics for two to three years of growth, and keep a written migration runbook instead of planning on --alter --partitions.
Alert on per-partition consumer lag and bytes in so a hot key shows up before it causes an outage.

Key takeaway: The partition key is the unit of order, parallelism, state and compaction in Kafka, so choose it as the smallest set of records whose order matters and keep it stable for the life of the entity. Make every producer hash the same way, size keyed topics for years rather than months, keep join inputs co-partitioned, and grow a keyed topic by migrating to a new one rather than adding partitions in place.