Title: Lecture 3 — Direct and Indirect Prompt Injection Attacks
Course: AIINFRA 302: AI Security, Guardrails & Governance > Section: Week 03 — Direct and Indirect Prompt Injection Attacks

📖 Lecture — Direct and Indirect Prompt Injection Attacks

In AIINFRA 300 and 301 you learned to build LLM apps and RAG pipelines by feeding the model instructions and context. This week we look at the same mechanism from the attacker's side: if you can steer a model by giving it text, so can anyone else who gets text in front of it. Direct vs. indirect injection. Simon Willison coined the term "prompt injection" in 2022, drawing a deliberate analogy to SQL injection: just as an app that concatenates untrusted user input into a SQL query lets an attacker rewrite the query, an app that concatenates untrusted text into a model's context lets an attacker rewrite its instructions. Direct prompt injection is the simple case — the attacker is the user, typing adversarial instructions straight into the chat box ("ignore your previous instructions and..."). Indirect prompt injection is the dangerous case for production systems — the malicious instructions are hidden in third-party content that a benign user's agent later ingests: a web page it browses, a PDF it summarizes, an email it reads, a code comment it processes. The user never sees or types the attack; their own assistant carries it in for them.

 | Direct injection | Indirect injection
Who supplies the malicious text | The end user, in the prompt | A third party, hidden in retrieved content
Does the victim user know? | Yes — they're the attacker | No — they're an unwitting carrier
Typical channel | Chat input box | Web pages, PDFs, emails, RAG documents, tool outputs
Why it's easy to miss | It's obviously adversarial text | It looks like ordinary content until the model "reads" it as instructions

The foundational research here is Greshake et al. (2023), "Not what you've signed up for," which demonstrated indirect injection hijacking Bing Chat and plugin-enabled assistants simply by planting instructions in web content the assistant retrieved. That paper's core insight still holds in 2025: any text the model reads is a potential instruction. A retrieved document doesn't arrive with a tag saying "this is data, not commands" — the model just sees tokens, and if those tokens read like instructions, a sufficiently unguarded model will often follow them. The lethal trifecta. Willison's 2025 framework gives you a practical design-review tool. An agent becomes exploitable via injection when all three of these are true simultaneously: (1) it has access to private data, (2) it is exposed to untrusted content, and (3) it has a way to communicate externally (send email, make an API call, post to the web). Remove any single leg and the exfiltration attack breaks — an agent that reads untrusted web pages but can't send data anywhere is annoying to attack; an agent with private data and outbound communication but no exposure to untrusted content has nothing to inject into. Use this as a checklist every time you design an agent: which of the three legs does this system have, and can I architecturally remove one? This is not theoretical. A 2025 multi-model study found 94.1% of 17 state-of-the-art models vulnerable to at least one injection vector — direct, RAG backdoor, or inter-agent communication. Documented 2025 case studies include an AI email assistant that exfiltrated private emails with zero user interaction (the poisoned email arrived, the agent read it, the agent acted), and a poisoned public document that leaked proprietary RAG data to anyone who asked the right question. These aren't edge cases dreamed up by researchers in a lab — they're the direct, predictable consequence of the lethal trifecta being present in shipped products. Multi-turn attacks. Not every attack is a single malicious message. Crescendo (Microsoft Research, USENIX Security 2025) is a multi-turn jailbreak that starts with completely benign conversation and progressively steers the model toward a prohibited objective, typically succeeding in under five turns. It works by exploiting the model's tendency to trust and build on its own prior outputs — each turn nudges the model a little further, using its own previous (still-innocuous-looking) response as leverage for the next escalation. This matters for guardrail design: a filter that only inspects each message in isolation will miss an attack that's only visible across the arc of a conversation. Correcting three common misconceptions. First, prompt injection is not limited to a malicious user typing an attack into the chat box — indirect injection hides payloads in content the end user never sees or types, which is precisely what makes it dangerous in production. Second, content retrieved by a RAG pipeline or returned by a tool is not automatically trusted data safe to feed straight into the model — it must be treated as untrusted input, segregated from developer instructions, exactly like unsanitized user input in a web app. Third, injected instructions are not limited to making a model say something embarrassing — in agentic, tool-connected workflows they can trigger real data exfiltration or unauthorized actions, as the email-assistant and poisoned-RAG cases above demonstrate concretely. The throughline for this course: once your model can read untrusted text and take actions, every document, webpage, and tool result your system touches is an attack surface you must design for, not an inert input you can trust by default.