Why architecture matters here
Redis Streams architecture matters because it fills a real niche. Kafka is overkill at moderate volume; SQS lacks ordering; SNS lacks replay. Streams gives you an ordered log with consumer groups + ack semantics.
Cost is Redis memory + optional persistence. Efficient.
Reliability comes from persistence (AOF) + cluster. Watch for eviction if MAXLEN forgotten.
The architecture: every command explained
Walk the diagram top to bottom.
Producer. XADD stream * field value ... Adds entry with auto-generated ID.
Stream key. Redis key holding the log. Append-only.
Consumer group. XGROUP CREATE creates group; XREADGROUP reads pending or new entries.
Entry ID. Millisecond-sequence (1234567890-0) monotonic within a stream.
Consumer names. Multiple consumers per group; work distributed.
XACK. Consumer ACKs after processing. Entry removed from pending.
XPENDING. Query in-flight entries per consumer/group.
XCLAIM. Reassign stuck entries from dead consumer to another.
MAXLEN trimming. Bounded length; older entries drop.
Persistence + Cluster. AOF for durability; Cluster shards by key.
End-to-end streaming flow
Trace usage. XADD orders * user_id 42 amount 100. Redis returns "1687432110001-0" as entry ID.
Consumer group "billing" created. Two consumers billing-a and billing-b.
XREADGROUP GROUP billing billing-a COUNT 10 STREAMS orders >. Returns 5 new entries.
Billing-a processes. XACK for each. Entries removed from pending.
Meanwhile billing-b also XREADGROUPs. Gets next batch.
Billing-a crashes mid-batch. Its entries remain in PENDING. XPENDING shows this.
Monitor detects; XCLAIM reassigns to billing-c.
Stream grows. Maxlen 1000000 set on XADD ~ 1000000. Old entries drop.
Persistence: AOF captures every XADD.
Cluster: stream key hash-slots to one shard; still supported.