Title: Lecture 1 — Why Production RAG Needs an Operations Lifecycle
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 01 — Introduction to Production RAG Architecture and the LLMOps Lifecycle

📖 Lecture — Why Production RAG Needs an Operations Lifecycle

Welcome to AIINFRA 301. In AIINFRA 300 you learned to build an LLM application. This course is about what happens after the demo works — when real users, real data, and real failure modes show up. We start with RAG because it is the single most common production LLM pattern, and because it forces a set of operational questions that a plain chatbot never does.

Why RAG exists

Large language models are frozen at training time and know nothing about your private documents. Ask a base model about your company's refund policy and it will either say "I don't know" or, worse, confidently invent an answer. This is the hallucination problem, compounded by two others: stale parametric knowledge (the model's internal facts stop updating the day training ends) and no access to private data (your internal wikis, tickets, and contracts were never in the training set). RAG addresses all three by grounding the model's output in retrieved evidence at the moment of the query, rather than trying to bake every fact into the model's weights. The core idea is a three-stage paradigm:

Stage | What happens | Why it matters
Retrieval | Convert the user's query into a search over an external knowledge store (usually a vector index) and pull back the most relevant passages | Determines whether the model even sees the right facts
Augmentation | Insert the retrieved passages into the prompt alongside the user's question | Determines whether the model can use what it sees
Generation | The LLM produces an answer conditioned on the augmented prompt | Determines whether the answer is faithful to the evidence

Get any one of these three wrong and the final answer suffers — which is exactly why we can't treat RAG as one black box.

The four layers of a production RAG system

A weekend prototype usually only implements the RAG chain. A production system needs four distinct layers, each with its own owner, its own failure modes, and its own tests:

1. Data pipeline — chunk, embed, and index your source documents. This is the least glamorous layer and, as you'll see below, the most consequential.
2. RAG chain — understand the query, retrieve candidates, augment the prompt, generate the response. This is the layer most tutorials stop at.
3. Evaluation and monitoring — continuously measure retrieval quality, answer faithfulness, latency, and cost, both offline (before shipping) and online (in production).
4. Governance and LLMOps — data lineage (where did this chunk come from, and can we trace an answer back to a source document?), access controls (does this user have permission to see this retrieved content?), and change management.

Notice that layers 3 and 4 don't exist in a naive prototype at all. That gap is the difference between "a demo that worked once" and "a system we can trust."

The LLMOps lifecycle: seven stages

LLMOps is the discipline of operating LLM-powered systems reliably over time. It extends classic MLOps but is reorganized around the realities of prompting and generation rather than training. The seven stages are:

1. Use case and model selection — decide the problem worth solving and which model/vendor fits the latency, cost, and quality bar.
2. Prompt engineering and versioning — treat prompts as versioned artifacts, not throwaway strings.
3. Deployment and monitoring integration — wire the app to logging/tracing infrastructure before it goes live, not after an incident.
4. Ongoing monitoring and observability — track live traffic: latency, token usage, error rates, drift in query patterns.
5. Evaluation and QA — run structured evals (which you'll build starting Week 9) against both new releases and live traffic.
6. Cost and latency optimization — caching, batching, model right-sizing, retrieval-count tuning.
7. Governance, compliance, and incident response — audit trails, PII handling, and a plan for when (not if) something breaks.

You'll notice this lifecycle is circular, not linear — production systems cycle back through prompt changes, re-evaluation, and re-deployment continuously.

RAGOps: LLMOps with a data-management focus

A 2025 paper, RAGOps: Operating and Managing RAG Pipelines (arXiv:2506.03401), argues that RAG systems need their own operational extension of LLMOps because the thing that breaks most often isn't the model — it's the retrieval corpus. RAGOps adds three practices on top of standard LLMOps:

- Continuous retrieval-corpus evaluation — don't just eval the model's answers; eval whether retrieval is pulling the right documents in the first place.
- Automated data-quality checks — catch corrupted, duplicated, or outdated source documents before they poison the index.
- The pipeline as the unit of operations — you monitor and version the whole pipeline (parser → chunker → embedder → index → retriever), not just the model endpoint.

Five pillars that distinguish LLMOps from classic MLOps

If you've done MLOps before (model training, versioning, CI/CD for models), here's what's genuinely new in LLMOps:

Pillar | Classic MLOps equivalent | What's different for LLMs
Prompt lifecycle management | Feature engineering | Prompts are versioned, tested, and rolled back like code
Evaluation and testing | Offline model metrics (accuracy, F1) | Needs LLM-graded and human-graded evals for open-ended text
Observability and tracing | Basic prediction logging | Full trace of every retrieval + generation step, token-by-token
Cost optimization | Compute cost at training time | Per-token, per-call cost that scales with every user request
Safety guardrails | Fairness/bias audits at training time | Real-time input/output filtering against a live, adversarial user base

Correcting three common misconceptions

Misconception 1: "A RAG app is just embed-retrieve-generate glued together, so the prototype loop is production-ready." Correction: production RAG is a multi-stage pipeline whose stages must be independently testable, measurable, and replaceable. If you can't swap your embedding model or re-run your chunker in isolation and measure the effect, you don't have a production system — you have a script. Misconception 2: "RAG retrieval is magic, so picking a good vector DB and LLM means the rest takes care of itself." Correction: the unglamorous fundamentals — data cleaning, parsing correctness, chunking strategy — matter most. A bad PDF parser that mangles tables or drops footnotes produces noise that no downstream vector database or LLM can recover from. Garbage in, garbage retrieved, garbage generated. Misconception 3: "Indexing the knowledge base once at launch means the RAG system is done." Correction: stale, ungoverned source data is a leading cause of production RAG failure. Documents change, policies get updated, products get discontinued — re-indexing and data governance are ongoing operations, not one-time setup tasks. This is precisely the gap RAGOps is trying to close.

Where this course is going

This week's lifecycle framing is the throughline for the rest of AIINFRA 301. Weeks 9–11 will drill into evaluation methodology, Weeks 12–13 into observability and tracing, Week 14 into prompt engineering and versioning, and Week 15 into production monitoring. Keep this week's diagrams handy — you'll be referencing them all term.