Title: Lecture 4 — Reproducibility, Data Versioning, and Planning for the Unexpected
Course: AIINFRA 303: Capstone Project > Section: Week 04 — Project Planning: Milestones, Risk, and a Reproducible Repository

📖 Lecture — Reproducibility, Data Versioning, and Planning for the Unexpected

By now your capstone project has a direction. This week we shift from "what am I building" to "how do I build it so that it actually holds together for the next eleven weeks" — and so that anyone, including future-you six months from now, can reproduce what you did and trust it.

Why reproducibility is not optional

Reproducibility means any experiment can be repeated exactly, and any production model can be traced back to the exact code, data, and configuration that produced it. In a capstone context this matters for a very practical reason: you will run dozens of experiments over the next several weeks, and if you can't tell which run produced which result, you can't debug regressions, defend your final numbers, or even remember why last week's model outperformed this week's. Reproducibility breaks in a handful of predictable ways:

Failure mode | What it looks like | Fix
Inconsistent library versions | "Works on my machine" — a teammate or grader installs requirements.txt and gets different numbers or a crash | Pin exact versions (==, not >=); use a lockfile
Undocumented preprocessing | A cleaning step, filter, or column drop lives only in someone's memory or a throwaway notebook cell | Preprocessing lives in versioned pipeline code, not ad hoc notebook cells
Unfixed random seeds | Train/test split, weight init, or sampling differs run to run | Set and log a seed for every library that uses randomness (numpy, torch, random, splitters)
Untracked data changes | Dataset gets "cleaned up" in place with no record of the before/after | Version the data itself, not just the code that touches it

None of these are exotic problems — they are the default outcome of moving fast without guardrails. The good news is that the guardrails are cheap to install and, once in place, run automatically.

The four practices that make reproducibility real

1. Seed fixation. Set a single seed value at the top of your config and thread it through every stochastic operation — data splitting, shuffling, initialization. Log it alongside your results.
2. Data versioning (e.g., DVC — Data Version Control). Git tracks code well but is a poor fit for large binary files like datasets and model weights. DVC tracks pointers to your data in Git while storing the actual bytes elsewhere (local cache or remote storage), so git checkout of an old commit also gets you the exact data that commit used.
3. Experiment tracking (e.g., MLflow). Every run's parameters, metrics, and artifacts get logged automatically instead of living in scattered spreadsheet rows or Slack messages.
4. Git tags marking release/milestone versions of code and models — git tag v0.1-baseline gives you a permanent, human-readable anchor point you can always return to, separate from the churn of ordinary commits.

Standardizing your repo layout

You don't need to invent a folder structure from scratch. Cookiecutter Data Science (CCDS) is a widely used template that gives you a sensible default layout — separate data/raw, data/processed, src/, models/, notebooks/, and reports/ directories, a Makefile or task runner, and a README.md skeleton built for exactly this purpose. Standardizing on a known layout means anyone who has seen the template before (including your future teammates, or a code reviewer) already knows roughly where to look.

Pipelines you can re-run end to end

The real test of reproducibility is this: can a stranger clone your repo and run one command to regenerate your results from raw data? DVC's pipeline feature (dvc.yaml) lets you define stages — prepare, train, evaluate — each with declared dependencies and outputs. Running dvc repro walks the dependency graph and re-executes only the stages whose inputs changed, end to end, from raw data to final metrics.

Enforcing hygiene automatically

pre-commit hooks run checks every time you attempt a commit, before the commit is allowed to complete. For this course, two are especially valuable:

- ruff — fast Python linting, catching style and correctness issues before they enter history.
- nbstripout — strips notebook output cells before commit, so your diffs stay readable and you don't accidentally commit large embedded images or, worse, output that leaked a credential or sensitive data sample.

Planning the work itself: milestones and risk

Reproducibility protects the integrity of your work; planning protects the delivery of it. Two lightweight artifacts do most of the work:

- A milestone roadmap: break the capstone into small, dated checkpoints rather than one big "finish project" deadline. GitHub Projects' Roadmap (timeline) view, combined with a Status field (On track / At risk / Off track), gives you and your instructor a shared, always-current picture of where things stand.
- A risk register: a short table of the top risks that could derail your project, each with a likelihood, an impact, and — critically — a mitigation.

Correcting common misconceptions

"My estimate is probably about right — I've thought it through." Most task estimates are best-case, gut-feel guesses, and best-case estimates are systematically optimistic. The fix isn't to guess harder; it's to decompose the work into small milestones and build your overall estimate from the sum of those smaller, more checkable pieces. "I wrote a risk register, so I'm covered." A risk register with vague contingencies ("we'll deal with it if it happens") almost never fires in practice, because there's no defined moment that triggers action. Every risk needs a concrete trigger condition — an observable event — paired with the specific mitigation that trigger launches. "If GPU credits run out before Week 8, switch to CPU-only quantized inference" is a trigger; "we'll manage compute carefully" is not. "I'll just add my dataset to .gitignore and I'm safe." .gitignore only prevents files from being tracked in the future. If a secret or a large binary is already committed, adding it to .gitignore afterward does nothing — it's already permanently in your Git history. Configure .gitignore before your first commit, and pair it with secret-scanning pre-commit hooks so accidental commits get caught immediately, not after the fact. "Git can just track my dataset like any other file." Git is not built for large binary files — every version of a large file you commit stays in the repository's history forever, bloating it permanently and making clones painfully slow. Keep large files out of Git proper; use .gitignore plus Git LFS or external storage (which is exactly what DVC manages for you). This week's lab puts all of this into practice on your own capstone repository — by the end, you'll have a reproducible foundation the rest of your project builds on.