Title: Quiz 12 — Observability, Evaluation, and Cost Instrumentation
Course: AIINFRA 303: Capstone Project > Section: Week 12 — Observability, Evaluation, and Cost Instrumentation

Q: A 200 OK response with sub-second latency guarantees that an LLM's answer was factually correct.
- True
- False
A: False

Q: Which of the following is considered the "fourth pillar" of observability for LLM-based systems, beyond logs, metrics, and traces?
- Load balancing
- Automated evaluation
- Container orchestration
- Database indexing
A: Automated evaluation

Q: Name one open-source, self-hostable tool covered this week that provides LLM tracing, prompt versioning, and evaluation without vendor lock-in.
Feedback: Sample/expected answer: Langfuse

Q: Which OpenTelemetry (OTEL) GenAI semantic convention attributes are used to capture token-level cost and performance data on a trace span?
- `http.status_code` and `http.method`
- `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`
- `db.query` and `db.rows_affected`
- `k8s.pod.name` and `k8s.namespace`
A: `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`

Q: Name the four RAGAS metrics used to quantitatively evaluate a RAG pipeline.
Feedback: Sample/expected answer: Faithfulness, answer relevancy, context precision, and context recall

Q: A single monthly provider bill is sufficient cost instrumentation because it shows total LLM spend.
- True
- False
A: False

Q: In this week's lab, which tool is used to allocate Kubernetes compute costs per namespace and per pod?
- Prometheus
- Grafana
- OpenCost
- RAGAS
A: OpenCost

Q: According to this week's lecture, what should you do with a real production example that fails an automated evaluation, in order to build a continuous-improvement loop?
Feedback: Sample/expected answer: Save it as a regression/eval test case so future changes are checked against it