Why it matters

Airflow is de facto ETL DAG. Understanding shapes integration.

Advertisement

The architecture

DAG in Python.

Operators: run tasks.

Java agent as containerized step.

Airflow + JavaPython DAGairflow-nativeKubernetesPodOperatorrun containerJava agentin podAirflow + K8s executor + container-based Java agents work well
Airflow.
Advertisement

How it works end to end

KubernetesPodOperator: run any container.

DockerOperator alt.

XCom: small data between steps.

Sensors: wait conditions.