Why architecture matters here
Workflow orchestration architecture matters because businesses depend on multi-step processes that must complete correctly even under failure. Payment plus fulfillment plus notification plus reconciliation — either all happen or the customer is unhappy. Without an orchestrator, teams reinvent the durability primitives poorly; with one, the guarantees are shared and consistent.
Cost is real but well-spent. Temporal servers cost money; workers cost money; storage for workflow history grows. But the alternative — bespoke retry logic, state machines in your database, cron-triggered recovery jobs — costs more in engineer time and outage exposure over any real time horizon.
Reliability is the entire point. Workflow engines give you exactly-once semantics for the whole workflow (given careful activity design). Restarts are transparent. Retries are automatic. Human intervention paths are first-class. The architecture is what makes this possible.
The architecture: every piece explained
Walk the diagram top to bottom.
Workflow Definition. Code that describes the process. In Temporal it is a function that calls activities and reacts to signals. In Airflow it is a DAG in Python. In Step Functions it is an Amazon States Language JSON. The definition is checked into git like any other code.
Scheduler / Server. The engine that runs the workflow. It persists the workflow state, dispatches activities to workers, records their results, and orchestrates retries. Temporal uses a Cassandra or Postgres backend for state; Airflow uses its metadata DB.
Workers. The processes that execute activities. Workers pull tasks from queues, run the code, and report results. Workers are stateless; they can be scaled horizontally.
Durable State. The workflow's execution history stored durably. Every activity result, every signal, every timer fires and event is appended. If a worker crashes, the workflow can be replayed from history on another worker without losing state.
Task Queue. Sharded queues for activities. Workers subscribe to specific queues based on their capabilities. Priority queues let high-priority workflows jump the line.
Retry + Timeout. Per-activity retry policy (exponential backoff, max attempts) and timeout. The engine handles all retries transparently; activity code doesn't need retry loops.
Signals + Queries. Runtime interaction. Signals push events into a running workflow ("user approved!"). Queries read workflow state without changing it ("what's the current status?"). Both enable rich interactive workflows.
Child Workflows. Workflows can invoke other workflows as sub-workflows. This composes cleanly and lets you break large workflows into manageable pieces.
Versioning + Migration. Long-running workflows may span code deploys. Temporal's patched() API and Airflow's DAG versioning let you evolve workflow code without breaking in-flight instances.
Observability. Web UI shows current state, past executions, activity durations, retries. Metrics stream to Prometheus; traces to your APM. Debugging is dramatically easier than raw code.
End-to-end signup workflow
Trace a signup flow. A user signs up. The signup service starts a "onboard_user" workflow.
The workflow function runs: `send_welcome_email` activity. Temporal dispatches to a worker, the worker sends the email via SES, returns success. Result recorded in history.
Next line: `await verify_email` — wait for a signal. Workflow pauses. The engine holds state; no worker is holding memory.
Two hours later, the user clicks the verification link. The signup service sends a signal to the workflow. Temporal wakes the workflow; workflow function receives the signal and continues.
`provision_account` activity dispatches. The worker calls the account service. Network hiccup; the activity fails. Retry policy kicks in: exponential backoff, up to 5 attempts. Attempt 2 succeeds. Result recorded.
`notify_sales` activity dispatches. Sales CRM is down for maintenance for 30 minutes. The activity fails; retries burn through. Eventually alerts the workflow team. A human resolves via workflow console — manually mark the activity done or restart from checkpoint. Workflow completes.
The whole thing survived worker restarts, network failures, and a downstream outage. Every step is auditable. Every retry is visible. The user experienced a normal signup.