Why architecture matters here
The architectural value of a queue is fate isolation, and it is worth being precise about what that buys. When the order service calls the fulfillment service synchronously, the two share every failure: a fulfillment deploy, a fulfillment GC pause, a fulfillment database stall all propagate backward into order-taking. Put SQS between them and the order service's write path depends only on SQS — a multi-AZ, replicated service with SLAs your own fleet cannot match — while fulfillment consumes at whatever rate it can sustain. Traffic spikes become queue depth instead of cascading timeouts; deploys become consumption pauses instead of dropped requests.
That decoupling has a price, and the architecture is how you pay it honestly. Latency becomes variable: work happens 'soon', not 'now', so queues suit workflows measured in seconds-to-minutes, not interactive reads. Duplicates become normal: at-least-once delivery means the consumer must make redelivery harmless — an idempotency key checked against a conditional write, or naturally idempotent operations. Teams that skip this discover it as double-charged customers. Failure handling becomes explicit: a message that keeps failing must not block or poison the queue forever, which is what redrive policies and dead-letter queues are for — they turn 'this one input crashes the consumer in a loop' from an outage into a quarantined artifact with an alarm on it.
SQS specifically wins on operational surface: no brokers to patch, no partitions to rebalance, no storage to size — the trade being fewer knobs than Kafka (no replay of consumed messages from the main queue, no consumer groups over a shared log, 14-day maximum retention). Choosing SQS is choosing a managed contract; the architecture below is that contract's machinery.
The architecture: every piece explained
Top row — the data path. Producers call SendMessage/SendMessageBatch (batching ten messages per call divides request cost by ten). The frontend authenticates, enforces per-queue quotas, and — for FIFO queues — applies deduplication: a content hash or explicit MessageDeduplicationId suppresses re-sends within a five-minute window, giving exactly-once enqueue even when producers retry. Accepted messages are written to replicated storage across multiple hosts and AZs before the send returns; a message acknowledged is a message durable. This distribution is also why a ReceiveMessage against a nearly-empty standard queue can return nothing — each receive samples a subset of storage hosts, which is exactly what long polling (WaitTimeSeconds up to 20) fixes by waiting and querying broadly instead of returning empty.
Consumers long-poll with batch receives (up to 10 messages). Receiving starts the visibility timeout: the message stays in the queue but is invisible to other consumers for the configured window (default 30s). The consumer processes and calls DeleteMessage; if it crashes or overruns, the timeout lapses and the message reappears for redelivery — this is the entire crash-recovery story, and heartbeat-style extension (ChangeMessageVisibility) is how long-running work keeps its lease.
Middle row — the semantics knobs. FIFO message groups: within one MessageGroupId, messages deliver strictly in order, one batch in flight at a time — so per-entity ordering (all events for order 123) is easy, and total throughput scales with the number of distinct groups, not consumers. Delay and retention: DelaySeconds hides new messages up to 15 minutes (scheduled work, debounce); retention up to 14 days bounds how long a stalled consumer fleet can lag before data loss. Redrive policy: each receive increments a counter; past maxReceiveCount the message moves to the dead-letter queue — same message, new address — where it waits for inspection and, post-fix, redrive back to source. Bottom: autoscaling keys off backlog metrics — ApproximateNumberOfMessagesVisible and, better, oldest-message age or backlog-per-instance — to size the consumer fleet to the arrival rate.
End-to-end flow
Trace an order through a fulfillment pipeline. Checkout completes; the order service sends a batch of events — OrderPlaced among them — to a FIFO queue with MessageGroupId=order-58211 and a deduplication ID derived from the event ID. The producer's SDK retries a transient 5xx; the dedup window swallows the duplicate enqueue. The send returns in single-digit milliseconds once the message is replicated across AZs. Checkout's latency budget never sees fulfillment.
A consumer instance long-polls the queue with WaitTimeSeconds=20, receives a batch of ten messages spanning several order groups (order-58211's messages arrive in exact order relative to each other), and starts a 90-second visibility timeout sized at roughly six times the p99 processing time. Processing order-58211 means reserving inventory and requesting a shipping label; the label API is slow today, so at the 60-second mark the consumer's heartbeat thread extends visibility by another 90 seconds. Inventory reservation is guarded by a conditional write on the event ID — if this exact event was already processed by a previous consumer whose delete never landed, the write no-ops and the consumer just deletes the message: idempotency making at-least-once invisible.
Now the failure path. A malformed address makes the label request throw; the consumer neither deletes nor extends beyond its current lease, the timeout lapses, and the message redelivers — receive count 2, then 3. At maxReceiveCount=5 the queue moves it to the DLQ, and order-58211's next messages unblock (a poison head no longer stalls the group forever). A CloudWatch alarm on DLQ depth pages the on-call; the payload shows the bad address, a validation fix ships, and the operator uses DLQ redrive to send the message back to the source queue, where it now processes cleanly — order recovered, no data lost, no queue drained by hand.
Meanwhile the evening spike triples arrivals. Backlog-per-instance crosses the target; the autoscaler grows the consumer fleet from 4 to 11; oldest-message age peaks at 40 seconds and recedes. Nobody was paged, and checkout never knew.