Order processing is the canonical event-driven workload: state changes pass through multiple services, must be exactly-once, must survive partial failures. Done well it's calm; done badly it's a 3am pager.
State machine first
Define states (PENDING → AUTHORIZED → SHIPPED → DELIVERED) and allowed transitions. Each transition emits an event. Stored in event log; current state is fold of events. Replayable for debugging.
Idempotency keys everywhere
Order ID + transition type = idempotency key. Re-delivered events deduped at the consumer. Critical: payment service must dedupe; shipping must dedupe. Test by replaying events.
Outbox pattern
Service writes state change + outbox event in same DB transaction. Separate process publishes outbox to Kafka. Atomic state + event publication without 2PC across DB and message broker.