Title: Lecture 15 — From Commit to Monitored, Cost-Aware Production
Course: AIINFRA 101: Applied Infrastructure & Containerization > Section: Week 15 — Automated Model Deployment, Monitoring & Cost Optimization

📖 Lecture — From Commit to Monitored, Cost-Aware Production

Every tool this course has built toward — Docker, Git, GitHub Actions, testing, containers, registries — chains together into one idea this final week: an automated pipeline that takes a code change to a live, monitored, cost-controlled production service, with a safety net if something goes wrong.

Blue-green vs. canary: pick based on confidence, not preference

Blue-green deployment runs two identical environments — "Blue" (live) and "Green" (new) — and flips all traffic from Blue to Green at once once Green passes validation. It is simple and near-instant to roll back. Canary deployment instead routes a small slice of traffic (say 5%) to the new version, watches metrics through a hold period, and ramps up (5% → 25% → 100%) only if things look healthy — requiring real traffic-splitting infrastructure like a service mesh (Istio, Linkerd). The choice depends on how confident you are in offline validation, not which one "sounds more professional." A config-only change or a model thoroughly validated against production-like data is a good fit for blue-green. A model whose real-world behavior can't be fully pre-validated (new feature pipeline, shifted user population) belongs on canary, where a small slice absorbs the risk first.

The full pipeline, stage by stage

1. Code commit — a change (feature engineering, serving code, model artifact) is pushed to Git.
2. CI build/test — GitHub Actions builds and runs automated tests (unit tests, linting, pytest quality gates).
3. Containerize — the validated service is packaged into a Docker image, the pattern you've used since early weeks.
4. CD deploy (blue-green/canary) — the container ships to production using a strategy that limits blast radius, ideally behind a manual approval gate for the highest-risk step.
5. Production monitoring — predictions, latency, and resource usage are continuously tracked.
6. Auto-rollback — if monitored metrics degrade past a threshold, the pipeline reverts to the last-known-good version without waiting for a human to notice.

The payoff of automating this chain is repeatability: a human following a 20-step runbook eventually skips a step under pressure; a pipeline won't. One failure mode worth naming: pipeline consistency validation — verifying that the Git commit, the code, and the deployed artifact all actually correspond to each other before promotion — catches the "container built from an old commit" bug that doesn't crash anything, it just silently underperforms. (Teams doing heavier experiment tracking and model-lineage work should look at the MLflow-based Model Registry patterns covered in AIINFRA 100 and AIINFRA 301 — this course keeps the pipeline itself as the focus.) Not every team should automate every stage immediately. Google's MLOps maturity model runs from Level 0 (fully manual) through Level 1 (automated training, manual deploy) to Level 2 (the full loop automated end-to-end). More automation is not automatically "more mature" — a pipeline that auto-deploys without validation checks is a liability, not an achievement. The goal is to match automation level to your team's monitoring maturity, and to earn Level 2 deliberately rather than assume it's the only acceptable destination.

Watching the meter: observability and cost control

Once a service is live, two questions matter: is it working? and what is it costing? CloudWatch (or an OpenTelemetry-compatible equivalent) is built around three data types that solve three distinct problems: metrics (numerical time-series — GPU utilization, latency), logs (event records — "model load failed"), and alarms (rules that fire when a metric crosses a threshold). Anomaly detection applies ML to a metric's history to build a dynamic band that auto-adjusts for time-of-day and weekly patterns, so you're not hand-tuning a fixed threshold. Cost control follows a strict order of operations: rightsize first (match capacity to actual observed p95 usage), schedule non-production environments off nights/weekends, and only then commit to Reserved Instances or Savings Plans on the right-sized footprint. Committing to a 1–3 year discount before rightsizing just locks in a discount on waste. GPU workloads deserve special scrutiny — utilization across production deployments typically runs only 30–50%, and hidden costs like data egress and fast storage don't show up until you look at the itemized bill. Cost optimization is a continuous weekly loop (alert → triage → action → verify), not a one-time setup task — the same discipline as monitoring, aimed at a different question.

Correcting the misconceptions

- Blue-green and canary are not interchangeable "flavors" of the same thing — picking the wrong one for a poorly-validated change risks exposing 100% of users with no gradual warning signs.
- A deployment pipeline without a tested rollback plan is incomplete, not a finished pipeline missing a nice-to-have.
- "More automation always means more mature MLOps" is false. Maturity is measured by what a pipeline catches (training-serving skew, delayed labels, data drift) — not by how few humans touch it.
- "I should lock in a discount plan early to save money faster" backfires. Rightsize first, on real data, then commit.