📖 Lecture — Documenting and Measuring Fairness in ML Systems

Up to now in this program you've focused on building things that work — RAG pipelines that retrieve the right chunks, agents that call the right tools. This week we turn to a harder question: work for whom? A model can hit 95% aggregate accuracy and still fail one demographic group badly, and a single headline number will never tell you that. This week is about the documentation and measurement practices that surface those failures before they reach production — and about being honest with yourself and your stakeholders when they do. Model Cards. In 2019, Margaret Mitchell and colleagues at Google proposed Model Cards for Model Reporting as a standardized documentation format for trained models. A model card is not a marketing sheet — it's closer to a nutrition label. The framework asks you to report: intended use cases and explicitly out-of-scope uses (what the model should not be used for); the training data and its provenance; evaluation results disaggregated across relevant demographic or subgroup slices, not just one aggregate number; and a section on ethical considerations, known limitations, and risks. The disaggregation requirement is the heart of the idea. A face-verification model with 99% aggregate accuracy might be 99.7% accurate on light-skinned men and 65% accurate on dark-skinned women — a gap invisible in the topline metric and only visible once you slice the evaluation by subgroup. Model cards make that slicing mandatory, not optional. Datasheets for Datasets. Timnit Gebru and colleagues made a parallel argument one layer down the stack: datasets need documentation too, and bias frequently originates there rather than in the model architecture. A Datasheet for Datasets asks questions like: Why was this dataset created, and by whom? How was the data collected, and from what population? What preprocessing was done? What are the known gaps or skews in who and what is represented? If your training data underrepresents a group, no amount of clever modeling downstream fixes that — the model will underperform on that group because it barely saw examples of it. Datasheets and model cards are companion documents: one covers the fuel, the other covers the engine. Fairness metrics — and why there is no single right one. Once you decide to measure fairness, you need a metric, and this is where things get philosophically uncomfortable. Three commonly used definitions:

Here's the uncomfortable part: except in trivial cases (e.g., the base rates are already equal across groups, or the classifier is perfect), demographic parity and equalized odds are mathematically incompatible — satisfying one generally means violating the other. This isn't a tooling limitation or something a smarter algorithm fixes; it's a theorem. That means choosing a fairness metric is a values decision, not a purely technical one. Your team has to decide, for your specific context and its stakes, which kind of fairness you are optimizing for, and be able to justify that choice to auditors, users, and yourselves. "Just remove the protected attribute" doesn't work. A very common instinct is to drop race, gender, or other protected attributes from the training features and assume the resulting model is fair by construction. This is called fairness through unawareness, and it is a well-documented failure mode. The problem is proxies: ZIP code correlates with race in a segregated housing market; first name can correlate with gender or ethnicity; shopping patterns, browsing history, and dozens of other innocuous-looking features can jointly reconstruct the very attribute you removed. Modern gradient-boosted trees and neural networks are extremely good at finding these correlations even when no single feature is an obvious proxy. Removing the column doesn't remove the signal — it just makes the bias harder to see and audit. Tools that actually implement this. You don't have to compute any of this by hand. Fairlearn (Microsoft-backed, open source) provides fairness metrics, mitigation algorithms like ExponentiatedGradient (in-processing) and ThresholdOptimizer (post-processing), and an interactive dashboard for comparing models. IBM's AI Fairness 360 (AIF360) is broader still: 70+ fairness metrics and 9 mitigation algorithms spanning all three intervention points — pre-processing (fix the data, e.g. reweighing), in-processing (fix the training objective, e.g. adversarial debiasing), and post-processing (fix the predictions, e.g. reject-option classification). Aequitas offers a CLI/API/web-based bias audit workflow aimed at policy and civil-rights-style audits. And for the LLM/RAG systems you've been building in this program specifically, Giskard (Apache-2.0 licensed) automatically scans ML, LLM, and RAG applications for bias, data leakage, hallucination, and prompt injection — its RAGET component even auto-generates test questions to evaluate whether your retrieval pipeline is accurate across different query types. The throughline for this week: fairness isn't a checkbox you tick once. It's a metric you choose deliberately, a measurement you disaggregate honestly, and a set of tradeoffs you document in writing — so that the next person who touches your model (including future you) knows exactly what it does well, what it doesn't, and who might be affected either way.