Title: Lecture 15 — Reproducible, Tested, Tracked, and Responsible AI
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 15 — Reproducibility, Testing & Responsible AI — Final

📖 Lecture — Reproducible, Tested, Tracked, and Responsible AI

This is the closing week of the course, so it doubles as a synthesis. Across the term you wrote code that worked — you ran it, it produced the right answer, and you moved on. But "it worked when I ran it" is not the same as "it will keep working, for someone else, six months from now." Professional AI/ML practice is largely about closing that gap. This lecture walks the four habits that do it: (a) reproducibility and the ML lifecycle, (b) testing and code quality, (c) experiment tracking, and (d) responsible-AI governance.

Part A — Reproducibility and the ML development lifecycle

Every AI system has to be built, tested, and updated by real people, over and over, for years. The discipline that makes that manageable is MLOps (Machine Learning Operations): it takes practices software engineers already rely on — automation, version control, testing, monitoring — and applies them across the ML lifecycle: preparing data, training, evaluating, deploying, and watching behavior in production. Without it, teams end up with "science projects" — a model that worked once, on someone's laptop, that nobody can rebuild later. (Deep MLOps and production monitoring are owned by AIINFRA 101/301; here we cover just enough to build the habit.)

Why is ML reproducibility harder than ordinary software? In traditional software, if you have the source code you can rebuild the app. In ML, the result depends on three moving parts: the code that defines training, the data it trained on, and the environment (exact versions of scikit-learn, PyTorch, Python itself). If any one drifts, the model you get back can differ — subtly — from the one you started with. Two beliefs feel intuitive but will bite you: saving a model.pkl is not enough (a pickle captures weights but not what data, code, or library versions produced them), and re-running a notebook on your own laptop is not proof of reproducibility (it only shows the code runs here, now). True reproducibility requires pinned dependencies, versioned data, and ideally a controlled environment. Data-versioning tools like DVC exist for the "data" leg; this course focuses on the code and environment legs.

The most common symptom is the oldest line in software: "but it worked on my machine!" It is especially common in AI work, where a script leans on a dozen libraries, each with its own version. For years the Python fix was pip plus a requirements.txt: install packages, then pip freeze > requirements.txt to dump versions, and teammates run pip install -r requirements.txt. It works, but requirements.txt is often a loosely-versioned list (e.g. numpy>=1.20) that doesn't reliably capture transitive dependencies — the packages your packages depend on — so two people can resolve subtly different environments weeks apart.

This is exactly what uv (the fast Rust-based manager from Week 9) solves with a lock file. The first time you run uv run, uv sync, or uv lock, uv resolves every dependency — direct and transitive — to an exact version and writes uv.lock. Anyone who runs uv sync then gets the identical set of versions, byte-for-byte. Is uv.lock just a fancier requirements.txt? Not quite — the single most common point of confusion. Think of requirements.txt as a hand-editable shopping list and uv.lock as a tool-managed, itemized sealed receipt: you can read the TOML, but you don't hand-edit it — uv regenerates it whenever dependencies change. If a pipeline only understands the classic format, uv export --format requirements.txt produces one from your lock.

| | requirements.txt (classic) | uv.lock (modern) | |---|---|---| | Generated by | pip freeze | uv lock / uv sync / uv run (automatic) | | Format | Plain text, one package per line | Tool-managed TOML | | Versioning | Often loose ranges; may miss transitive deps | Exact, fully resolved versions of all deps | | Hand-editable? | Yes, commonly edited | Not intended to be hand-edited | | Reproducibility | Good, but can drift | Guaranteed — same input, same result |

Locking is only half of deployment-readiness; the other half is project structure. A loose folder of .py files is not a package — no declared dependencies, no entry point, no metadata. The modern fix is a pyproject.toml at the project root declaring name, version, and dependencies in one tool-readable place. uv init scaffolds it; uv add <package> adds a dependency to pyproject.toml and updates uv.lock. That turns a folder of scripts into a real project — one you can hand to a teammate or wrap in the Docker image from Week 14 as the first COPY pyproject.toml uv.lock ./ step. And one last reproducibility reflex from the notebook world: Restart & Run All. Re-running top-to-bottom in a fresh kernel is the cheapest way to catch hidden state that makes a notebook "work" only because of the order you happened to run cells.

Part B — Testing and code quality for AI

"It ran" is not "it's correct, and stays correct." pytest is the most widely used Python test tool, popular because it removes friction: it auto-discovers any file named test_*.py containing functions named test_*, and inside them you use plain Python assert — no self.assertEqual(a, b) ceremony. If assert normalize([1, 2]) == [0.0, 1.0] is false, pytest shows expected vs. actual. When you notice repeated setup (the same toy dataframe or trained model in every test), @pytest.fixture gives you reusable setup that any test requests just by naming it as a parameter — which keeps ML tests short and DRY.

For AI/ML code, testing means more than a return value. Useful things to assert:

| What to test | Example assertion | |---|---| | Output shape | preprocess(X).shape == (100, 10) | | Output dtype | preprocess(X).dtype == np.float32 | | Determinism | predict(x, seed=42) == predict(x, seed=42) | | Edge cases | normalize([]) doesn't crash and returns [] | | Value ranges | 0.0 <= normalize(values)[i] <= 1.0 |

For style and cleanliness, Ruff is a single, extremely fast (Rust) tool that replaces the old Flake8 + Black + isort stack. Two commands you'll use constantly: ruff check . (linting — unused imports, undefined names, style issues) and ruff format . (auto-formatting to a consistent style). Both configure from one [tool.ruff] section in pyproject.toml, so it's fast enough to run on every save or commit. One misconception worth correcting: **once tests pass, the code is not "done."** Passing tests proves the code was correct at that moment, for the cases you tested. The real value comes from running tests and linters continuously — every change, every commit — to catch regressions, where a fix for one thing quietly breaks another. Testing is not a finish line; it's an ongoing safety net.

Part C — Experiment tracking with MLflow

Once a project is locked and packaged, a natural question follows: as you tweak parameters, how do you remember what you tried and what worked best? That's MLflow Tracking — the next layer on top of packaging, and a required topic for this course. With a few API calls you log parameters (learning rate, number of trees), metrics (accuracy, loss), and even the trained model, each time you run. Wrap training in with mlflow.start_run():, call mlflow.log_param(...) and mlflow.log_metric(...), and MLflow saves each run to a local store. Then mlflow ui launches a local dashboard (by default at http://localhost:5000) where every run sits side-by-side for comparison. MLflow also has a model registry that lets a team formally mark a version "Staging" or "Production" — a far more deliberate act than copying a .pkl to a server. Don't believe the myth that tracking is only for big production teams: even a single learner running three or four variations benefits enormously from logging instead of relying on memory or sticky notes.

Put together, that's a deployment-readiness checklist: (1) pin and lock deps with uv; (2) package with pyproject.toml; (3) track experiments with MLflow; then (4) containerize (Week 14). Each layer removes a little more "it worked on my machine" risk.

Part D — Responsible and ethical AI

Just because you can deploy an AI system doesn't mean you should — or that you should under any conditions. Responsible AI isn't a bolt-on checklist; it's habits and safeguards designed in from day one, like security or reliability. (This is a survey; the topic is covered in depth in AIINFRA 302.)

The NIST AI Risk Management Framework (AI RMF 1.0), released January 2023, is voluntary but has become the shared U.S. vocabulary for AI risk. It rests on four interconnected functions that loop continuously across a system's life:

| Function | Core question | Example activity | |---|---|---| | Govern | Who is accountable, and what's our culture around AI risk? | A sign-off process before a model ships | | Map | What could go wrong, for whom, in what context? | Identifying who's affected by a hiring-screening model | | Measure | How do we quantify and track those risks? | Running bias metrics or red-teaming a chatbot | | Manage | What do we actually do about them? | Adding human review, retraining, or rolling back |

The goal is trustworthy AI — valid, reliable, safe, secure, accountable, transparent, explainable, privacy-enhanced, and fair. "Trustworthy" is not the same as "accurate": a model can ace a test set and still be untrustworthy if it's opaque, biased, or has no accountable owner. NIST keeps evolving this — the 2024 Generative AI Profile (NIST-AI-600-1) names risks like confabulation ("hallucination") and data leakage — so treat governance as a living practice, not a one-time checklist.

The EU AI Act takes a different tack: sort systems into risk tiers with obligations that scale to the risk — unacceptable (banned, e.g. government social scoring), high-risk (heavy obligations: assessments, documentation, human oversight — e.g. hiring or credit-scoring tools), limited-risk (transparency/disclosure — e.g. chatbots, deepfakes), and minimal-risk (largely unregulated — e.g. spam filters). Its deadlines are a moving target — they have already shifted more than once — so the lesson isn't to memorize dates but to keep a process for tracking changes. Two misconceptions to correct: the Act applies extraterritorially (like GDPR — if your system's output is used in the EU, you can fall under it regardless of where you're based), and removing protected attributes does not remove bias. Proxy variables — zip code, first name, shopping history — can reconstruct race or gender the model never directly saw; a résumé screener that never sees "gender" but weights a women's professional association is still discriminating. Removing bias means actively Measuring outputs across groups, not just scrubbing columns. Bias itself has three common origins — pre-existing (already in the world and captured in data, as in the COMPAS recidivism tool), technical (design choices in features, labels, objectives), and emerging (only visible once deployed). The core insight: AI doesn't invent bias; it automates and scales whatever bias already exists.

That's the course in one arc: build it, then make it reproducible, tested, tracked, and responsible. Those are the habits that separate a demo from a system you can trust.