📖 Lecture — Provenance, Signing, and the AI Supply Chain

By now you've built LLM apps and RAG systems in AIINFRA 300 and 301. You know how to pull a model off Hugging Face, drop it into a pipeline, and get it running in minutes. This week asks an uncomfortable question: how do you actually know that model is what it claims to be, and that it's safe to load? The honest answer, for most of the AI ecosystem today, is that you don't — not without doing some extra work. That extra work is what "supply-chain security" means in an AI context, and it borrows heavily from lessons the software industry already learned the hard way with open-source packages. Why downloading a model is not like downloading data. The biggest misconception to clear up first: a model file is not inert data the way a CSV or a JSON file is. For years, the default way to save a PyTorch model was Python's pickle format. Pickle doesn't just store numbers — it stores instructions for reconstructing Python objects, and those instructions can include arbitrary code that executes the moment the file is deserialized (i.e., the moment you load() it). A malicious actor can hand-craft a pickle file that looks like a normal model checkpoint but silently runs code — installing malware, exfiltrating credentials, or opening a backdoor — the instant a researcher or student loads it into memory. This isn't theoretical; malicious pickle-format models have been discovered on public model hubs. Loading a model file is running code, not just reading data. Safetensors: fixing the format, not just scanning for bad actors. The response the ML community converged on is a new serialization format called safetensors, now an audited, PyTorch Foundation–governed project. Safetensors stores only tensor data (the actual weight values) with a simple header — there is no mechanism for embedding executable instructions at all. It's not that safetensors is "scanned more carefully"; it structurally cannot execute code on load, because the format has no concept of code. As of early 2026, roughly 42% of Hugging Face models are tagged safetensors, which means the majority still ship in formats that can carry a code-execution payload. When you have a choice, prefer safetensors, and treat any pickle/.bin/.pt checkpoint from an unverified source with real suspicion. A model card is documentation, not proof. The second misconception: "the model card tells me where the model came from, so I've done my provenance homework." A model card is a human-written description — often accurate, sometimes outdated, occasionally aspirational. It is not cryptographically bound to the actual weight file you downloaded. Nothing stops someone from uploading a modified or poisoned model under a legitimate-looking card, or from a card simply going stale after the underlying weights are updated. Real provenance means being able to verify, mathematically, that a specific artifact came from a specific, identity-verified source and hasn't been altered since. That's a job for signing, not prose. Sigstore: bringing keyless signing to model weights. The software supply-chain world already solved a version of this problem for containers and packages using Sigstore — a toolchain of three components. Cosign signs and verifies artifacts. Fulcio issues short-lived certificates tied to an OIDC identity (like a GitHub or Google login) instead of requiring you to manage long-lived private keys. Rekor is a public, tamper-evident transparency log that records every signing event, so anyone can later verify that a specific artifact was signed by a specific identity at a specific time. The OpenSSF Model Signing SIG is now adapting this exact stack to model weights, so a "signed model" can mean the same verifiable thing a "signed container image" already means in DevOps. Popularity is not vetting. The third misconception is the most seductive: "this model has 2 million downloads, surely someone would have caught a problem by now." Public hubs host more than 500,000 models, and the overwhelming majority have never been through a formal security audit. Download counts measure popularity and convenience, not scrutiny. OWASP's LLM Top 10 codifies this under LLM03 (Supply Chain Vulnerabilities): third-party models, fine-tuned adapters, datasets, and plugins all need their own inventory and verification, precisely because "everyone uses it" is not a security control. Documenting what you can't see: SBOM, ML-BOM, and the G7/CISA push. You can't manage risk in components you can't enumerate. That's the logic behind the Software Bill of Materials (SBOM) idea, now extended to AI. In 2025–2026, CISA and G7 partners released shared guidance defining seven "clusters" of elements that producers and consumers of AI systems should document for transparency — building on a June 2025 shared G7 vision for AI supply-chain visibility. CycloneDX has operationalized this for machine learning specifically with the ML-BOM (or AI-BOM): an extension of the standard SBOM that also enumerates base models, training and fine-tuning datasets, evaluation benchmarks, and their dependencies.

The throughline for this week: provenance is a stack, not a single artifact. A model card gives you a story. Safetensors removes one entire class of risk. Sigstore-style signing gives you cryptographic proof of origin. An ML-BOM tells you everything else you'd need to check. None of these alone is "the answer" — together, they're how you move from trusting a hub's reputation to actually verifying what you're deploying.