Why architecture matters here
ML fails on operations, not on algorithms. The model that got 0.94 F1 in the notebook now serves at 0.72 because the feature store diverged from the training pipeline. The model that shipped is now hallucinating because label drift went unnoticed. The regulator asks for lineage and no one can produce it.
The architecture matters because every one of these failures is preventable with the right substrate. A feature store enforces training/serving parity. A model registry provides lineage. Monitoring catches drift. Governance produces audit trail.
With the pieces in place, the model is one small thing inside a large, reliable system.
The architecture: every piece explained
The top strip is the data-to-model flow. Data lake holds raw and curated data (Iceberg or Delta commonly). Feature store exposes both offline (batch, historical) and online (low-latency) feature views; a shared definition ensures training and serving see the same values. Training runs distributed with experiment tracking; every run logs code, data snapshot, hyperparameters, and metrics. Model registry stores versioned model artifacts with metadata and approval workflow.
The middle row is the serving and observation loop. Serving exposes online (real-time) and batch (offline scoring) inference. Shadow / canary splits traffic to compare new vs current model without user impact. Monitoring tracks input drift, prediction distribution, latency, and business quality (when labels arrive). Feedback loop writes labels back to the data lake for the next training cycle.
The lower rows are the substrate. Governance holds lineage (which data → which model), approval workflow, and audit. Compute plane allocates GPU and CPU with quotas per team. Metadata + orchestration — MLflow, Kubeflow, Airflow — orchestrates DAGs and stores metadata so every model in production is traceable back to its raw data.
End-to-end flow
End-to-end: a fraud team registers a new feature — transaction velocity per user last 24h. The feature store materializes it in both offline (nightly batch to Iceberg) and online (streaming to Redis) modes with a shared definition. A training run reads offline features, trains a gradient-boosted model, and logs run metadata. The model artifact is registered as v42 with metrics. Approval workflow requires a data-science lead to sign off. Deployment routes 5% shadow traffic to v42; monitoring shows fraud recall +3% with acceptable latency. Canary promotes to 50%. Monitoring holds. Full promotion to 100%. Two weeks in, monitoring detects input drift on velocity; the feature engineer investigates; a retraining is triggered from the same DAG. Every step has lineage.