🛠️ Lab 12 — Instrument, Cost, and Evaluate Your Capstone (50 pts)

Goal: Add production-grade observability, cost allocation, and RAG evaluation to your own capstone service, all running locally with free and open-source tools, producing a live Grafana dashboard plus a RAGAS evaluation notebook that establishes your project's quality baseline. Steps:

  1. Instrument your FastAPI service. In your capstone repo, add prometheus-fastapi-instrumentator to your API service and wire it up to expose a /metrics endpoint. Confirm it's live by curling the endpoint locally and checking that request count, latency histograms, and in-progress request gauges are present.
  2. Stand up Prometheus and Grafana locally. Use Docker Compose (or your local minikube cluster from earlier weeks) to run Prometheus and Grafana as containers. Configure Prometheus to scrape your FastAPI /metrics endpoint on an interval (e.g., every 5s).
  3. Reference Jeremy Jordan's ml-monitoring repo. Clone the free, open-source ml-monitoring reference implementation and compare its Prometheus/Grafana wiring to yours — use it as a template for dashboard panels (latency percentiles, request rate, error rate) rather than starting from a blank Grafana dashboard.
  4. Build your Grafana dashboard. Import or hand-build panels for: request throughput (req/s), p50/p95/p99 latency, error rate, and in-flight requests. Point every panel at your own capstone's live traffic (generate synthetic load with a simple script if needed).
  5. Add OpenCost for cost allocation. Install OpenCost on your local Kubernetes cluster (minikube is fine) following the OpenCost documentation. Confirm you can view cost allocation broken down by namespace and by pod for the namespace running your capstone.
  6. Build a RAGAS evaluation set. Assemble a small eval set (10–20 question/answer/context triples) drawn from your own capstone's RAG or LLM layer — real questions a user of your project would plausibly ask.
  7. Run the RAGAS evaluation. In a notebook, run RAGAS against your eval set and compute faithfulness, answer relevancy, context precision, and context recall. Record the scores as your project's baseline — this is the number future weeks' changes will be measured against.
  8. Close the loop. Pick at least one RAGAS example that scored poorly, add it as a saved regression test case in your eval set, and write two sentences on what you'd change in your retrieval or prompt to fix it.

Deliverables: A screenshot (or short screen recording) of your live Grafana dashboard showing real traffic from your capstone service, a screenshot of your OpenCost namespace/pod cost breakdown, and your RAGAS evaluation notebook (.ipynb or exported .pdf) with baseline scores and your written note on the one failing example you're carrying forward.