Cloud Composer architecture — Airflow, DAGs, workers, scheduler

Why architecture matters here

Composer fails on DAG parse errors, worker overload, and connection sprawl. Architecture matters because scheduler + workers + metadata + connections all need capacity + governance.

The architecture: every piece explained

The top strip is the runtime. DAG files. Scheduler. Workers (K8s). Metadata DB.

The middle row is composition. Web UI. Operators BQ/Dataflow/K8s. Sensors + branches. Connections + secrets.

The lower rows are ops. IAM + audit. Metrics. Ops — upgrades + capacity + retries.

End-to-end flow

End-to-end: DAG defines nightly ETL. Scheduler parses at midnight; queues tasks. Workers execute BQ + Dataflow operators. On failure retry with backoff. Web UI shows success. Metadata DB retains history.