Title: Lecture 14 — Docs-as-Code: Runbooks, Reproducibility, and Model Cards
Course: AIINFRA 303: Capstone Project > Section: Week 14 — Documentation, Runbooks, and Reproducibility

📖 Lecture — Docs-as-Code: Runbooks, Reproducibility, and Model Cards

You have spent thirteen weeks building a real system: containers, cloud infrastructure, CI/CD pipelines, inference serving, fine-tuning, agents, RAG, security controls, and cost management. All of that engineering has value only if someone besides you — a teammate, an auditor, a future employer evaluating your portfolio, or you yourself in six months — can understand it, run it, and fix it when it breaks. This week is about closing that gap. We are going to treat documentation as a first-class engineering deliverable, not an afterthought you write the night before a demo. Playbooks vs. runbooks. These two words get used interchangeably in casual conversation, but in operations they mean different things, and the distinction matters. A playbook is strategic: it describes what a team should do and why, often covering a class of situations ("how we handle any security incident") with room for judgment calls. A runbook is tactical: it is a documented, step-by-step set of procedures that guides an engineer through one specific operational task or incident ("the inference API is returning HTTP 500s — do these six things in this order"). Runbooks exist to standardize incident response so that the outcome doesn't depend on which engineer happens to be on call. A good runbook directly reduces MTTA (mean time to acknowledge) because the on-call engineer isn't guessing where to start, and it reduces MTTR (mean time to resolution) because every step has a known, verifiable expected output. Think of the playbook as the chapter in the manual and the runbook as the specific checklist taped to the machine. What goes in a runbook. At minimum, a runbook for an incident like "service is returning 500s" should include: the symptom and how it's detected (which alert fires, which dashboard panel goes red), a triage sequence (check logs, check upstream dependencies, check recent deploys), specific commands to run with their expected output, a rollback procedure, and an escalation path if the steps don't resolve it. The table below maps the core sections you should include this week.

Runbook section | Purpose | Example content
Trigger / symptom | How you know this runbook applies | "p99 latency \> 2s for 5 min" or "5xx rate \> 1%"
Diagnosis steps | Ordered, executable checks | docker logs api --tail 100, check GPU utilization, check upstream API status
Expected output per step | Lets the operator confirm they're on track | "Healthy: \<50ms avg. Unhealthy: timeouts or connection refused"
Remediation | The fix once root cause is found | Restart container, scale replicas, roll back to last known-good image tag
Rollback procedure | How to undo a bad deploy | git revert, redeploy previous tagged image, restore previous model version from registry
Escalation | Who/what to contact if unresolved | Contact info, or "page the on-call" placeholder for this solo capstone

Why reproducibility is a documentation problem, not just a tooling problem. Back in Week 4 you were introduced to experiment tracking. The discipline matters again here because reproducibility is fundamentally about someone else being able to trace a deployed model back to the exact code, data, parameters, and metrics that produced it. MLflow gives you the mechanism: a local tracking server (backed by SQLite, with a local artifact store) logs every run's parameters, metrics, and artifacts, and the Model Registry lets you promote your best run to a named, versioned model. When your README says "we deployed model v3," MLflow is what lets someone click through from that version number to the exact run, code, and metrics that justify it existing. The Model Card. Reproducibility tells someone how to recreate your model. A Model Card tells someone whether they should use it. Following the Hugging Face Model Card template (which you saw applied to responsible-AI questions back in Week 11), your card should state the model's intended use and out-of-scope uses, the training data and its known gaps, the evaluation metrics and the conditions under which they were measured, and known limitations or fairness concerns. This is not bureaucratic box-checking — it's the artifact that lets a downstream user avoid using your model for something it was never validated to do. Now, the misconceptions. First: a runbook that's too long or too complex won't be followed under pressure. This is one of the most common documentation failures in the industry. During a real outage, an on-call engineer under stress will not read five paragraphs of prose to find the one command they need. If a runbook isn't skimmable in the first ten seconds, it gets abandoned and the engineer improvises — which is exactly what the runbook was supposed to prevent. Keep every step short, specific, executable, and paired with an expected output so the operator can self-verify without guessing. Second: automating a process before you've done it manually is a common cause of broken runbooks. It's tempting to jump straight to a script that "handles the incident for you." But if you haven't performed the recovery by hand first, you don't yet understand every edge case, and your automation will confidently do the wrong thing at 2 a.m. Do it manually, write it down as you go, run it by hand a second time to confirm the steps are complete and correct, and only then consider scripting the parts that are truly mechanical. Third: code doesn't speak for itself. Even flawless, well-commented code cannot tell a reader what problem it solves, how to set up the environment, how to actually run it, or what output to expect. That is the README's job, and no amount of clean code substitutes for it. Every project — including your capstone — needs a README that states the problem, the setup steps, the run command, and the expected result. This week you'll practice all of it at once: log a reproducible run in MLflow, register the model, write the Model Card that explains it, write the runbook that lets someone operate it, and publish a documentation site that ties it all together.