Title: Midterm Exam
Course: AIINFRA 302: AI Security, Guardrails & Governance > Section: Week 08 — PII Protection with Presidio and Output Validation

📝 Midterm Exam (100 pts)

Instructor note: the answer key below is hidden from students in the Canvas quiz build; remove or lock this section before publishing the student-facing version.

Part A — Applied Scenario (40 pts)

You are building a customer-support assistant for a mid-sized insurance company. The assistant retrieves policy documents via RAG, answers customer questions about coverage, and can draft emails to customers on the agent's behalf. In Weeks 1–7 you learned how to design prompts, build a RAG pipeline, evaluate retrieval quality, and reason about threat models for LLM applications. Scenario: During a demo, a customer support rep pastes a full customer complaint email (containing the customer's name, policy number, and a partial SSN) into the assistant to ask "summarize this and draft a reply." The assistant's reply includes the customer's full SSN, restated verbatim, in the drafted email text. Question: Using what you've learned about RAG architecture, prompt design, and LLM threat modeling (Weeks 1–7), explain (1) at least two distinct points in this pipeline where this failure could have been caught before reaching the rep, and (2) how you would redesign the pipeline's data flow to prevent this specific failure from recurring. Be specific about where in the request/response flow each control sits. → A strong answer identifies: (1) input scanning — the pasted complaint email should have been scanned/anonymized before being included in the prompt, since it contains raw customer PII; and (2) output scanning — even if the SSN entered the prompt, the drafted reply should have been scanned before being shown to the rep or sent, since output must be treated as untrusted per LLM05. A redesigned flow: incoming text → PII analyze/anonymize → assemble prompt with placeholders → LLM call → scan output for PII/regex patterns → block or redact matches → only then surface to the rep or send. This also ties to earlier weeks' point that RAG systems must treat all retrieved and user-supplied content as untrusted, not just the system prompt.

Part B — Weeks 1–7 Review (60 pts, 5 pts each)

1. (MC) Which of the following best describes the purpose of a system prompt in an LLM application?

a) To store conversation history b) To set persistent instructions and constraints that guide the model's behavior across a session c) To encrypt user data before sending it to the model d) To replace the need for a retrieval step → b

1. (TF) True or False: In a RAG pipeline, the retriever's job is to generate the final answer text.

→ False — the retriever fetches relevant chunks; the generator (LLM) produces the final answer.

1. (Short answer) Name two common chunking strategies used when preparing documents for a vector store.

→ Fixed-size chunking and semantic/recursive chunking (accept: sentence-based, paragraph-based, or overlap-based chunking).

1. (MC) Which metric is most directly used to evaluate whether a RAG system retrieved the relevant chunks for a query?

a) BLEU score b) Recall@k / precision@k c) Perplexity d) Token count → b

1. (TF) True or False: Prompt injection attacks can only occur through the initial user message, never through retrieved documents.

→ False — injected instructions can be embedded in retrieved/RAG content just as easily as in user input.

1. (Short answer) What is the main risk OWASP LLM01 (Prompt Injection) describes?

→ An attacker crafts input (direct or indirect, e.g., via a document) that manipulates the LLM into ignoring its intended instructions or performing unintended actions.

1. (MC) Which of the following is an example of a threat modeling exercise for an LLM application?

a) Measuring model latency b) Identifying trust boundaries and what happens if an attacker controls a specific input surface c) Choosing a chunk size for embeddings d) Selecting a vector database → b

1. (TF) True or False: Embeddings from different embedding models are generally interchangeable in the same vector index.

→ False — embeddings from different models are not directly comparable/compatible; mixing them degrades retrieval quality.

1. (Short answer) Why is it important to version and evaluate your RAG pipeline (retrieval + generation) rather than just the LLM prompt alone?

→ Because retrieval quality, chunking, and embedding choices materially affect the final answer; evaluating only the prompt misses failures introduced upstream in retrieval.

1. (MC) Which of the following is the best reason to keep an LLM's system prompt separate from user-supplied content in your application code?

a) It reduces token costs b) It preserves a clearer trust boundary and makes it harder for user input to override instructions c) It is required by the OpenAI API d) It improves embedding quality → b

1. (Short answer) In a threat model for an LLM app, what does "trust boundary" mean?

→ A boundary marking where data changes from more-trusted (system-controlled) to less-trusted (user- or externally-controlled), which is where validation/controls should be enforced.

1. (TF) True or False: A well-designed RAG system should assume that anything in the retrieved context could be adversarial or incorrect, not just anything typed by the user.

→ True.