Title: Lecture 4 — Layering Defenses Against Prompt Injection
Course: AIINFRA 302: AI Security, Guardrails & Governance > Section: Week 04 — Prompt Injection Defenses and Input Validation

📖 Lecture — Layering Defenses Against Prompt Injection

Last week you learned how easily prompt injection can hijack an LLM's behavior — the Gandalf challenges made that visceral. This week we shift from attack to defense. The single most important idea to internalize up front comes straight from OWASP's LLM Prompt Injection Prevention Cheat Sheet: given the stochastic nature of LLMs, no fool-proof prevention is known to exist. That is not a temporary gap waiting for a patch — it is a structural property of how these models work. Every defense you learn this week is a mitigation, not a cure. Your job as an engineer is to stack enough mitigations that the residual risk becomes acceptable for your use case, and to build monitoring so you notice when something slips through. Layer 1: Spotlighting. Microsoft's spotlighting research (arXiv 2403.14720) is the cheapest, most accessible first layer, and it directly targets a root cause of injection: the model can't reliably tell the difference between "instructions I should obey" and "data I should merely process." Spotlighting makes that boundary explicit through three techniques:

Technique | What it does | Example
Delimiting | Wraps untrusted content in clear markers so the model treats it as data, not instructions | <<<UNTRUSTED_DOCUMENT_START>>> ... <<<UNTRUSTED_DOCUMENT_END>>>
Datamarking | Interleaves a marker character/token throughout the untrusted text so even reformatted or partially-quoted excerpts still "look" untrusted | Replacing spaces in the pasted document with a rare token like ^
Encoding | Transforms untrusted text (e.g., base64) before it reaches the model, then decodes only for final output, so embedded instructions can't be parsed as instructions mid-context | Base64-encoding retrieved web content before insertion into the prompt

Spotlighting measurably lowers injection success rates in Microsoft's own testing. But notice the framing: it is "necessary but not sufficient." It's the cheapest layer, not the last one. An attacker with enough creativity can still find phrasing that survives a delimiter or convinces the model to "step outside" the marked boundary. Layer 2: Instruction hierarchy. OpenAI's instruction hierarchy work trains the model itself to weigh instructions differently depending on where they came from: system instructions outrank developer instructions, which outrank user messages, which outrank tool output and retrieved content. This is a model-level defense — it lives in training, not in your application code — and it's meant to be combined with the app-level delimiting from Layer 1, not to replace it. Crucially, OpenAI describes this as degrading gracefully: the hierarchy makes injected instructions from low-trust sources less likely to be obeyed, but it does not guarantee they'll never succeed. Think of it as raising the bar, not building a wall. Layer 3: Architectural defenses. If spotlighting and instruction hierarchy are about making the model behave better, architectural defenses are about limiting the blast radius when it doesn't. Two patterns matter here:

- Dual LLM pattern: split your system into a privileged LLM that plans and acts (has tool access) and a quarantined LLM that only processes untrusted content and has no tool access at all. Even if the quarantined LLM gets fully "hijacked" by injected instructions, it has no tools to misuse — it can only pass text back, which the privileged LLM treats as untrusted data (tying back to Layer 1's spotlighting).
- CaMeL (Google DeepMind, 2025, arXiv 2503.18813) goes further with capability-based policies: actions are only permitted if they're backed by an explicit capability grant, so even a fully injected instruction stream cannot trigger an unauthorized action, because the system, not the model's good judgment, is what enforces the boundary.

Putting it together: a practical defense-in-depth stack. In production, you layer all of this:

1. Delimit untrusted content (spotlighting).
2. Rely on instruction hierarchy as a model-level backstop, not a guarantee.
3. Validate and constrain inputs (scanners like LLM Guard, structured validation like Guardrails AI).
4. Restrict tools to the minimum needed for the task.
5. Require human approval for sensitive or irreversible actions.
6. Separate planning from execution (Dual LLM / capability-based policies).

Tools you'll use in the lab. LLM Guard (pip install llm-guard) is Protect AI's open-source scanning library; its input_scanners.PromptInjection scanner uses a fine-tuned classifier (ProtectAI/deberta-v3-base-prompt-injection-v2) to flag likely injection attempts before they ever reach your model. Rebuff (pip install rebuff) layers heuristics, LLM-based detection, a vector database of known attacks, and canary tokens (secret strings planted in a prompt that let you detect if the model's output "leaked" internal instructions back to an attacker) — but Rebuff's own documentation is explicit that it is a prototype and "cannot provide 100% protection." Guardrails AI takes a different angle: Guard.for_pydantic() validates that structured output conforms to a schema, catching cases where injected instructions successfully changed what the model tried to do. Guardrails AI's February 2025 "Guardrails Index" benchmarked 24 guardrails against each other — worth a skim if you want to see how wide the quality gap is between tools that sound similar on paper. Correcting three common misconceptions. First: "A keyword blocklist or regex filter on inputs will stop prompt injection." It won't, reliably. Encoding tricks (base64, leetspeak, Unicode homoglyphs), multilingual obfuscation (translating the malicious instruction into a language your filter doesn't cover), and roleplay framing ("pretend you are DAN, an AI with no restrictions...") routinely slip past static filters. Filters catch the attacks people already know about; they don't generalize to novel phrasing. Second: "There is a technical fix that fully eliminates prompt injection." We opened this lecture with OWASP's position precisely because this misconception is so common and so consequential: no such fool-proof prevention is known to exist. If a vendor claims their product "solves" prompt injection, that claim should raise your skepticism, not lower it. The honest framing is layered mitigation plus containment plus monitoring. Third: "If I validate the input, I don't need to check the output." Because injection can't be fully blocked at the input stage, output-side scanning is equally necessary — checking for data leakage, policy violations, or signs that an unsafe action is about to be taken. Input validation and output validation are complementary, not redundant; a robust system needs both, plus the architectural containment from Layer 3 in case both get bypassed.