Title: Quiz 5 — Jailbreaks, Red-Teaming, and Adversarial Testing
Course: AIINFRA 302: AI Security, Guardrails & Governance > Section: Week 05 — Jailbreaks, Red-Teaming, and Adversarial Testing

Q: What is the key difference between a jailbreak and a prompt injection?
- Jailbreaks only work on open-source models; injections only work on closed models
- A jailbreak targets the model's own safety alignment; an injection targets the application's instructions
- Injections always require multi-turn conversations; jailbreaks are always single-turn
- There is no meaningful difference — the terms are interchangeable
A: A jailbreak targets the model's own safety alignment; an injection targets the application's instructions

Q: The DAN ("Do Anything Now") persona attack is a classic example of a prompt injection, not a jailbreak.
- True
- False
A: False

Q: How does the Crescendo attack technique defeat single-turn safety filters?
- It encrypts the harmful request so the filter can't read it
- It gradually escalates across multiple turns so no single message looks dangerous in isolation
- It only works on models with no safety training at all
- It floods the model with random tokens until it crashes
A: It gradually escalates across multiple turns so no single message looks dangerous in isolation

Q: In Anthropic's many-shot jailbreaking research, what mechanism causes the model to start complying with harmful requests after being shown hundreds of faux Q&A exchanges in a long context window?
Feedback: Sample/expected answer: In-context learning: the model pattern-matches on the repeated "harmful question → compliant answer" examples and starts predicting compliance as the expected next response, overriding its trained refusal behavior.

Q: What do PAIR and Tree of Attacks with Pruning (TAP) have in common?
- Both require white-box access to model weights and gradients
- Both use an attacker LLM and an evaluator/judge LLM in an iterative loop, with only black-box access to the target
- Both are manual, human-only prompt-crafting techniques
- Both only test for data leakage, not jailbreaks
A: Both use an attacker LLM and an evaluator/judge LLM in an iterative loop, with only black-box access to the target

Q: garak is a paid, closed-source vulnerability scanner that requires an NVIDIA GPU license to run.
- True
- False
A: False

Q: Which statement best reflects why red-teaming must be continuous rather than a one-time pre-launch activity?
- Because red-teaming tools charge a subscription fee that resets monthly
- Because models, prompts, tools, and attacker techniques keep evolving, so a clean scan today says nothing about tomorrow
- Because regulators require a new scan every fiscal quarter regardless of any changes
- Because red-teaming results expire automatically after 30 days
A: Because models, prompts, tools, and attacker techniques keep evolving, so a clean scan today says nothing about tomorrow

Q: Explain why a successful red-teaming session (i.e., one that finds no vulnerabilities) cannot be used as proof that an AI system is safe.
Feedback: Sample/expected answer: Red-teaming can only demonstrate the presence of vulnerabilities that were found within the scope, probes, and session tested — it cannot prove the absence of vulnerabilities that weren't tested for or don't yet exist. A clean result is evidence about that specific scope only, not a guarantee of overall safety.