AWS Kinesis Architecture in Depth

Why architecture matters here

Kinesis architecture matters because it's opinionated on things Kafka leaves flexible. Shards are the throughput unit; consumers scale via enhanced fan-out; delivery via Firehose is one-click. The trade-off is less control.

Cost is per shard-hour + PUT payload. For steady moderate volume Kinesis is competitive; at very high volume MSK may be cheaper.

Reliability is strong; AWS handles replication + regional failover.

The architecture: every service explained

Walk the diagram top to bottom.

Producer. Uses PutRecord API or Kinesis Producer Library (KPL) for batching.

Kinesis Data Stream. The stream. Composed of shards; retention configurable.

Consumer. KCL (Kinesis Client Library) manages sharding + checkpointing. Or Lambda triggers.

Shards. Throughput unit: 1 MB/s write and 2 MB/s read + 1000 records/s write. Add shards to scale.

Retention. 24 hours default; up to 365 days paid.

Enhanced Fan-Out. Dedicated 2 MB/s per consumer; multiple consumers don't share throughput.

Firehose. Kinesis Data Firehose — one-click delivery to S3, Redshift, OpenSearch, HTTP endpoints. Handles batching + retry.

Managed Flink. Streaming SQL on Kinesis streams. Serverless-ish.

Auto-scale streams. Adjust shard count; keeps ordering within shard.

vs MSK Kafka. Managed simplicity vs Kafka flexibility.

End-to-end streaming flow

Trace a workflow. App emits events to Kinesis via KPL. KPL aggregates + compresses; sends PutRecords.

Stream has 10 shards. Partition key hashes to shard; ordered writes within shard.

Consumer: KCL library on 5 EC2 instances. Each takes 2 shards; checkpoints to DynamoDB.

Second consumer added later (analytics). Enhanced fan-out: gets its own dedicated 2 MB/s per shard; doesn't compete with first consumer.

Firehose subscribes: batches every 5 minutes or 5 MB; writes Parquet files to S3.

Managed Flink app runs SQL: "SELECT COUNT(*) FROM stream GROUP BY user_id TUMBLE 1 MINUTE." Results emit to another stream.

Volume spike; auto-scale adds 5 shards. Existing consumers continue, but only by way of the parent-drain rule described in the section on hot shards and resharding below - a split closes the parent shard, and a consumer must read it to end-of-shard before touching either child.

The shard is the unit of ordering and of throughput

A Kinesis Data Stream is not one log. It is a set of shards, and the shard is the only object in Kinesis that carries a capacity number: 1 MB/s and 1000 records/s of ingest, 2 MB/s of egress shared across classic consumers. Stream capacity is nothing more than shard count multiplied by those figures. That part is familiar. The part that catches teams out is that the shard is simultaneously the unit of ordering. There is no stream-wide sequence. Each shard is an independently ordered log with its own monotonically increasing sequence numbers, and records in different shards have no defined relative order at all - not by arrival time, not by anything.

Those two roles are welded together and you cannot separate them. Wanting more throughput means wanting more shards, and more shards means your ordered groups get finer. Wanting a coarser ordering guarantee means fewer, larger ordered groups, which means fewer shards, which caps throughput. Every serious design decision on a Kinesis stream is a negotiation between those two facts.

If you take one thing from this article, take that: shard count is not a capacity knob you can turn freely, because turning it changes the ordering semantics your consumers observe. A stream sized purely from a throughput spreadsheet, with no thought given to what the partition key means, will work in load tests and produce hard-to-reproduce ordering bugs in production.

Partition keys and how records land on a shard

Every record you put carries a partition key - an arbitrary UTF-8 string you choose. Kinesis runs MD5 over that string to produce a 128-bit integer. The stream's key space is that full 128-bit range, and each shard owns a contiguous, non-overlapping slice of it, described by a HashKeyRange with a StartingHashKey and an EndingHashKey. The record lands in whichever shard's range contains the hash. That is the entire routing algorithm: no coordinator, no lookup table, no rebalancing of already-written data. The mapping is a pure function of the key and the current shard layout.

Two consequences fall straight out. First, the same partition key always maps to the same shard for as long as the shard layout is unchanged, which is exactly what makes per-key ordering possible. Second, the balance of your stream is determined by the distribution of your keys, not by the number of records. Ten shards with a uniform key distribution give you ten evenly loaded logs. Ten shards where most traffic carries tenant=acme give you one shard doing most of the work and nine idling, while the stream-level metrics look perfectly healthy.

ExplicitHashKey is the escape hatch: it lets a producer name the hash directly and bypass MD5, so you can place records deliberately rather than statistically. It is rarely the right tool, but it is the only way to pin a record to a chosen shard without engineering the key string to hash where you want it.

Ordering is per shard, not per stream

Kinesis guarantees this and only this: for records accepted into a given shard, a consumer reads them back in acceptance order, identified by increasing sequence numbers. It guarantees nothing across shards, and nothing about the relationship between the order your producer called the API and the order the service accepted the calls.

That guarantee is narrow enough that key choice is the design. The rule: the partition key must be the identity of whatever thing needs its events serialised. If the consumer applies a state machine per order, the key is the order ID. If it maintains a running balance per account, the key is the account ID. Pick something coarser than the entity - a region, a fixed constant, a small enum - and you buy ordering you did not need plus hot shards you did not want. Pick something finer, or random, and the guarantee evaporates: your consumer must then tolerate seeing an update before the insert it depends on, which usually means the consumer needs its own reordering buffer or a tolerant upsert.

Note also what the guarantee does not survive. A producer retry after an ambiguous timeout appends the payload a second time under a new sequence number. Concurrent producers writing the same key have no coordination between them, so "acceptance order" is whatever the network delivered. Ordering in the stream is not ordering at the producer, and if the true order matters, put a monotonic version or an event timestamp in the payload so the consumer can detect and drop stale updates rather than trusting arrival order alone.

Hot shards, resharding, and parent-child lineage

When one key or one narrow band of the hash space takes a disproportionate share of traffic, that shard throttles while the stream as a whole sits well under capacity. Adding shards does not necessarily fix it. A split divides a hash range, so if all the heat sits on a single hash value, one child inherits the entire problem. The fix for a genuinely hot key is upstream: salt the key across N buckets (acme#0 ... acme#7) and accept that ordering is now per bucket rather than per tenant, or give that entity its own stream.

Resharding itself is two operations. SplitShard takes one shard plus a hash key inside its range and produces two children covering the two halves. MergeShards takes two shards with adjacent ranges and produces one child covering the union. In both cases the parent is not deleted - it is closed. A closed shard accepts no further writes but keeps serving reads of everything already in it until retention expires, and its SequenceNumberRange gains an EndingSequenceNumber.

The rule resharding imposes on consumers

That closure is precisely what lets ordering survive a reshard, and it imposes a hard rule: drain the parent before starting either child. A consumer must read a closed shard all the way to end-of-shard - the point where GetRecords returns a null NextShardIterator - and only then pick up the children, which name their ancestry through ParentShardId and AdjacentParentShardId. Start a child early and you will process a record for some key while an earlier record for that same key is still sitting unread in the parent, which breaks ordering exactly across the reshard boundary.

The KCL enforces this for you: a child shard's lease is not handed out until the parent's lease is checkpointed as SHARD_END. Hand-rolled consumers built directly on GetRecords routinely miss it, and that is the classic origin of "our ordering broke once, during a scale-up, and we could never reproduce it". Scaling is therefore not free of consumer-visible effects: for a short window after a split the consumer fleet is still draining parents, and iterator age on those parents can spike before the children come online.

Provisioned versus on-demand capacity

In provisioned mode you name the shard count and pay per shard-hour plus per PUT payload unit, regardless of how much of that capacity you use. You get exact control over the shard layout, which matters if you have engineered the key space deliberately, and you own scaling: resharding is your API call, typically driven by an alarm on incoming bytes or records measured against the per-shard limits.

On-demand removes shard count from your interface. The service tracks throughput and reshards on your behalf, and you pay per GB written and read rather than per shard-hour. The shards are still there - they still appear in ListShards, ordering is still per shard, and consumers still have to follow parent-child lineage across the splits the service performs for you. On-demand is a billing and control-plane change, not a change in semantics. Anyone who assumes on-demand means "no shards to think about" will be surprised the first time a hot key throttles one of them.

The choice usually comes down to traffic shape. Spiky or not-yet-measured traffic suits on-demand: you avoid provisioning for a peak you have never seen and you avoid writing scaling logic. Steady, well-understood, high-volume traffic is usually cheaper provisioned, because you are paying for capacity you genuinely use around the clock. Either way, on-demand scaling reacts to observed load, so a step change faster than it can adapt still throttles. It is elastic, not instantaneous, and a genuinely bursty producer still needs client-side buffering and retry.

Two consumer models - shared polling versus enhanced fan-out

Classic consumers pull. They call GetRecords against a shard iterator and receive a batch, and the shard's 2 MB/s of read capacity is shared across every classic consumer registered on it. Two consumers means roughly half each in practice. There is also a per-shard cap on GetRecords calls per second, which a tight polling loop with several consumers will hit before it hits the byte limit. This model costs nothing beyond the shard itself, and it is entirely adequate when you have one or two consumers and your latency is dominated by your own poll interval rather than by any service constraint.

Enhanced fan-out inverts the flow. You register a consumer against the stream with RegisterStreamConsumer, then call SubscribeToShard; the service pushes records to you over a persistent HTTP/2 connection, and that consumer gets its own dedicated 2 MB/s per shard that no other consumer touches. Latency drops because there is no poll interval - records are delivered as they arrive rather than when you next ask. You pay per consumer-shard-hour plus per GB retrieved, on top of the stream.

When enhanced fan-out earns its cost

Three situations justify it. You have more than two or three consumers on the same stream and they are starving each other of read bandwidth. You have a latency target that a polling loop cannot meet. Or you have one consumer whose lag must be insulated from another consumer's bad day - an alerting pipeline that cannot fall behind because the batch analytics job is replaying history.

It is not justified for a single consumer with a relaxed latency budget: that is paying a per-shard, per-consumer premium for throughput you already had. And note the cost multiplies shard count by consumer count, so it grows fastest exactly when you scale out. A common middle path is to leave low-priority consumers on shared polling and register only the latency-sensitive one for fan-out.

The KCL: lease table, checkpoints, and lease stealing

The Kinesis Client Library exists because the raw API hands you a per-shard iterator and nothing else - no work assignment, no progress tracking, no failover. The KCL adds all three, and it keeps that state in a DynamoDB table it creates in your account, named after your application. One row per shard. Each row holds the shard ID, the current lease owner, a lease counter, and a checkpoint: the sequence number of the last record the owner confirmed as processed.

Assignment works through lease renewal. Every worker periodically bumps the lease counter on the rows it owns. A worker that dies stops bumping. After the lease-expiry interval another worker notices the counter has not moved, takes ownership with a conditional write against the counter value it last observed, and resumes from the checkpoint in that row. DynamoDB's conditional put is the distributed lock: it is what stops two workers claiming the same shard, and it is why the lease table is not an optional convenience but the actual coordination substrate.

Lease stealing and rebalancing

Each worker computes how many leases it ought to hold given the number of active workers and the total shard count. A worker holding fewer than its share takes a lease from the most heavily loaded worker, one at a time, until the distribution evens out. This is what makes adding an instance to a consumer fleet a non-event: you start the process, it steals leases, work redistributes, and no operator action is needed. It also means a deploy that rolls instances produces a burst of lease handoffs, which matters for the duplicate-delivery discussion below.

Two things to plan for. The lease table is real capacity you pay for and it can throttle: many shards plus an aggressive renewal interval generates constant writes, and lease-table throttling presents as consumers repeatedly dropping and reacquiring shards in a loop, which looks like a Kinesis problem and is a DynamoDB one. And checkpoint frequency is a genuine tradeoff - checkpoint every record and you pay a DynamoDB write per record; checkpoint once per batch and a crash replays that whole batch.

At-least-once delivery, retention, and replay

Kinesis is at-least-once, never exactly-once, and the paths that produce a duplicate are structural rather than bugs. A producer that times out and retries writes the payload twice under different sequence numbers. A consumer that processes a batch and dies before checkpointing will, on lease handoff, replay from the last checkpoint - the new owner has no way to know how far past that checkpoint the previous owner actually got. An enhanced fan-out subscription re-established after a network fault resumes from the last acknowledged position, which can overlap what you already saw.

Therefore consumers must be idempotent, and that has to be designed in rather than hoped for. In practice it means one of: a natural idempotency key in the payload plus a dedup store with a TTL longer than your retention window; writes that are inherently idempotent, such as upserts keyed by entity ID and SET semantics rather than INCREMENT; or a sink that dedups for you. The record's sequence number is a stable identifier for a specific record in a specific shard and works well as a dedup key for the replay case - but not for producer-side double writes, which arrive as genuinely distinct records.

Retention is a replay window, not a cleanup setting

Records live in the stream for 24 hours by default and can be extended up to a year, and the crucial property is that reading does not consume them. That makes retention a first-class capability rather than housekeeping. You can point a brand-new consumer at TRIM_HORIZON and rebuild a derived store from scratch, or reset an existing application's checkpoints to an AT_TIMESTAMP position and reprocess yesterday after shipping a fix - without touching producers and without asking anyone to resend. Every hour of retention you buy is an hour of recovery you are able to perform. Set it from your worst-case recovery requirement, not from the default.

Operating a stream: iterator age, throttling, aggregation

Iterator age is the metric that matters

GetRecords.IteratorAgeMilliseconds is worth more than every other Kinesis metric combined. It is the gap between now and the timestamp of the last record handed to the consumer - that is, how far behind real time you are running. A healthy stream sits near zero. A rising iterator age means the consumer is losing ground, and if it keeps rising past your retention window, records expire unread and that data is simply gone.

Watch it per shard and never only in aggregate. The single most common Kinesis incident is one shard's iterator age climbing while the stream average looks fine: a hot key concentrates traffic on one shard, or one worker is wedged on a poison record, and the average buries a shard that is hours behind. Alarm on the maximum across shards.

Throttling on both sides

On the produce side, exceeding a shard's 1 MB/s or 1000 records/s yields ProvisionedThroughputExceededException from PutRecord. For PutRecords the failure is quieter and more dangerous: the batch call returns success while individual records fail inside the response, so a producer that only checks the HTTP status silently drops data. Always inspect FailedRecordCount and retry only the failed subset, with backoff and jitter.

On the consume side, exceeding the shared 2 MB/s or the call-rate limit throttles GetRecords, which slows the consumer, which raises iterator age, which tempts you to poll harder, which throttles more. That feedback loop is why a consumer that suddenly falls behind sometimes never recovers on its own, and why the correct first response to rising lag is usually to back off and add shards or fan-out rather than to increase poll frequency.

Aggregation and the per-record cost

Produce-side billing is per payload unit of a fixed size, so a firehose of small records pays for mostly-empty units and exhausts the 1000 records/s limit long before it touches the 1 MB/s one. The KPL's aggregation packs many user records into a single Kinesis record, cutting both the per-record cost and the record-count pressure, and the KCL de-aggregates transparently so consumers still see the original records. The price is latency - the KPL buffers before sending - and blast radius, since one aggregated record that fails takes every user record inside it with it. Aggregation also interacts with key choice: records aggregated together must share a shard, so the KPL groups by the hash range, and a very skewed key distribution limits how much aggregation you actually get.

Failure modes worth recognising on sight

Iterator age climbing on exactly one shard: a hot key or a stuck worker, not a capacity problem - adding shards without fixing the key will not help. Checkpointing before processing rather than after: converts at-least-once into at-most-once and loses records silently on every crash, and it is an easy mistake to make when the checkpoint call is placed at the top of the record loop for convenience. Duplicate delivery clustered around deploys: entirely normal, because rolling instances triggers lease handoffs and each handoff replays from the last checkpoint - it is evidence your consumer is not idempotent, not evidence that Kinesis is broken.

Where Kinesis sits next to Kafka and SQS

Against Kafka the shape is the same: shards are partitions, sequence numbers are offsets, and the KCL lease table plays the role of the consumer group coordinator. Kinesis trades Kafka's tunability - replication factor, log compaction, arbitrary consumer-group offset manipulation, broker configuration - for having no brokers to operate at all. If you want Kafka's semantics on AWS, that is Amazon MSK.

Against SQS the difference is more fundamental: SQS is a queue, not a log. Messages are deleted once acknowledged, so there is no replay and no second consumer reading the same message, and ordering exists only within a FIFO message group. See Amazon SQS for the visibility-timeout lease model, redrive and dead-letter queues.

The axis that actually decides it: if multiple independent consumers must read the same records, or you need to replay history, you need a log, and Kinesis is the low-operations log on AWS. If each unit of work goes to exactly one worker and is then finished with forever, a queue is simpler and cheaper. Reaching for Kinesis to build a work queue means paying for shards, ordering and retention you will not use, and inheriting a resharding problem in place of simply adding consumers.