Why architecture matters here
Batch architecture matters because batch failures are compounding: today's failure poisons downstream tables; next day's job builds on the poison. Right architecture (idempotency, retries, backfill) recovers cleanly.
Cost is compute + orchestrator infra. Right-size instances; use spot where safe.
Reliability from quality gates. Bad data blocked at boundary, not consumed downstream.
The architecture: every piece explained
Walk the diagram top to bottom.
Trigger. Cron schedule; event from data lake; manual for backfill.
Orchestrator. Airflow, Prefect, Dagster, or Argo Workflows. Schedules + tracks.
Workers. Spark clusters, K8s jobs, cloud batch. Execute tasks.
DAG definition. Python code (Airflow/Prefect/Dagster) or YAML (Argo) declaring tasks + dependencies.
Idempotency. Tasks safe to rerun. Write to date partition; MERGE by key.
Retry policy. Exponential backoff; max retries; different behavior per exception type.
Backfill. Reprocess historical windows on demand or after schema change.
Data quality gates. Great Expectations, dbt tests. Block downstream if bad.
Observability. Gantt view of task durations; lineage across DAGs; SLO alerts.
SLA + freshness. Downstream depends on outputs by X; monitor + alert.
End-to-end batch flow
Trace a batch. Airflow DAG "daily_sales_report" runs at 2 AM.
Task 1: extract_orders — reads raw orders from operational DB; writes to date partition in lake. Idempotent (overwrites partition).
Task 2: transform_orders — Spark job; joins with product catalog; writes enriched partition.
Task 3: quality_gate — Great Expectations checks: row count within 10% of prior day; primary key uniqueness; null percentages. All pass.
Task 4: aggregate_by_region — SQL job. Task 5: publish_to_bi_warehouse.
Failure scenario: task 2 fails (spark cluster OOM). Retry policy: retry with double memory. Second attempt succeeds. Total: 20 min delay.
Backfill: schema change 3 days ago. Manual trigger for last 3 days; each day runs as separate DAG instance; parallelism configured.
SLA breach: quality gate fails one day. Downstream DAGs blocked. Alert sent; on-call investigates. Root cause: upstream feed had bad data.