Title: Lab 14 — Version, Label, and Roll Back a Production Prompt
Course: AIINFRA 301: Production RAG & LLMOps — Observability and Evaluation > Section: Week 14 — Prompt Management, Versioning, and Release Workflows

🛠️ Lab 14 — Version, Label, and Roll Back a Production Prompt (50 pts)

Goal: Move your RAG app's system prompt out of source code and into a Langfuse prompt registry, use labels to deploy specific versions without redeploying, and prove that your staging eval gate can catch — and let you instantly roll back — a bad release.

Steps

1. Stand up Langfuse for free. Use the free-tier Langfuse Cloud project you created in an earlier week, or self-host Langfuse locally with Docker Compose (both are free and documented in the resources below). Confirm you can log into the Langfuse UI and see the traces from your existing RAG app.
2. Create your first prompt in the registry. In the Langfuse UI (or via the Python/JS SDK), create a new prompt named something like rag-system-prompt containing the system prompt your RAG app currently has hardcoded. This becomes version 1.
3. Iterate to produce multiple versions. Make at least two more edits (e.g., add a citation instruction, adjust tone, add a refusal instruction for out-of-scope questions). Confirm in the UI that each edit creates a new version rather than overwriting the old one — check that version 1 is still visible and unchanged.
4. Apply labels. Assign the staging label to your latest version and the production label to an earlier, already-verified version. In your application code, replace the hardcoded prompt string with an SDK call that fetches the prompt by its production label at runtime (see the Langfuse "get started" and LangChain cookbook resources for the exact fetch pattern).
5. Confirm label-based deployment with no redeploy. Run your RAG app and confirm (via a log line, a printed prompt version ID, or a Langfuse trace) that it is using the version currently labeled production. Move the production label to point at your staging version — without touching or restarting your application code — and re-run a query. Confirm the app now uses the new version automatically.
6. Simulate a bad release. Create one more prompt version that looks like a "cleanup" but silently removes a real instruction (for example, delete the "always cite the source document" line). Label this version staging.
7. Run it through your Week 10/11 eval gate. Execute your existing evaluation script (or a small subset of your eval test set) against the RAG app while it's pointed at the staging label with the broken prompt. Capture the eval results and show that the regression is measurably worse (e.g., a citation-presence metric or faithfulness score drops).
8. Instantly roll back. Because the previous good version still exists untouched, reassign the production label back to the last known-good version ID. Re-run one query to confirm the app is serving the correct behavior again — with zero code changes and zero redeploys.
9. Write up your process. In 1–2 pages (or a README), document: your prompt's version history (a screenshot or table of version IDs, authors, timestamps), the labels you used and why, the eval results before and after the bad release, and the exact rollback step you took.

Deliverables: Submit your updated application code (showing the SDK fetch-by-label call), a screenshot or export of your prompt's version history from Langfuse, your eval-gate results comparing the good and bad prompt versions, and your 1–2 page write-up, via online upload or text entry.