Why architecture matters here
MLOps architecture matters because models drift, data changes, and business needs evolve. A model deployed once and forgotten silently degrades. A model deployed with monitoring + retrain automation stays fresh. The architecture is what separates the two outcomes.
Cost matters because training and serving compute add up. Efficient pipelines batch training, cache features, and serve with right-sized hardware.
Reliability comes from the eval + monitoring feedback loop. Offline metrics catch training regressions; online metrics catch behavioral regressions; drift monitors catch data-distribution shifts.
The architecture: every stage explained
Walk the diagram top to bottom.
Data Ingest. Streaming events + batch loads land data for training and features.
Feature Pipeline. Spark or Beam produces features. Same code path serves training and serving via feature store for symmetry.
Training. Distributed on GPUs; logs to experiment tracker; produces model artifact.
Model Registry. Versioned models with metadata (training data range, code version, offline metrics, lineage).
Offline Eval. Held-out sets, backtests. Model must pass gates to become candidate.
Serving. REST/gRPC for real-time; batch scoring for periodic use.
Online Eval. Shadow (traffic mirrored, response discarded), canary (small percentage), A/B (compare variants).
Monitoring. Feature drift (data changing), prediction drift (outputs changing), business metrics.
Retrain Triggers. Scheduled (weekly), drift-based (feature distribution shifted), event-based (new labeled data arrived).
Governance. Model cards + lineage + approvals + audit log across the whole cycle.
End-to-end model release flow
Trace a model release. Data scientist updates code + retrains. Training job runs; logs to MLflow. Produces model artifact.
Model registered as v1.7. Offline eval on golden set: AUC 0.87 vs baseline 0.85. All gates pass.
Deployment: canary at 5% traffic. Online eval compares v1.7 predictions vs v1.6. Business metric (conversion) marginally up. Ramp to 25%, then 100%.
Drift monitor watches: feature "avg_purchase_last_30d" has changed distribution (shopping season). Prediction distribution shifted slightly. Not a regression.
Two months later: drift is significant. Retrain trigger fires. Automated pipeline retrains on latest data. New candidate produced.
Offline eval passes; canary; ramp; full deploy. Old model kept in registry for rollback.
Governance: every model card in registry shows training data, code version, metrics, approval trail. Audit ready.