🛠️ Lab 14 — Trajectory Evaluation with DeepEval and GAIA (50 pts)

Goal: Write automated, LLM-as-judge-based test cases that score an agent's tool selection and answer relevance without labeled ground truth, then apply that evaluation approach to a slice of the real-world GAIA benchmark to practice trajectory- and tool-use evaluation on realistic, tool-requiring questions. Steps:

  1. Set up your environment locally, free-tier only. Create a fresh Python virtual environment and install the open-source DeepEval framework (pip install deepeval). You will need a judge model: use either a free-tier hosted API key you already have from earlier weeks, or a local model served through Ollama (e.g., a small instruction-tuned model) so the entire lab runs at no cost.
  2. Bring an agent to evaluate. Reuse the tool-using agent you built in an earlier week of this course (any framework is fine — the agent just needs to call at least one external tool per task). If you no longer have one handy, stand up a minimal agent with 2–3 tools (e.g., a calculator, a web search stub, a file reader).
  3. Write your first DeepEval test file. In Pytest style, create test cases that run your agent on 5 sample questions and score two things with LLM-as-a-judge metrics: (a) tool correctness — did the agent call an appropriate tool for the question, and (b) answer relevancy — does the final answer actually address the question asked. Note that neither metric requires a labeled "gold" answer — the judge reasons about appropriateness directly.
  4. Add a trajectory-level check. Extend at least one test case to grade the full trajectory rather than just the final answer: log every tool call and intermediate reasoning step, and write a custom LLM-as-judge metric (using DeepEval's custom metric support) with an explicit rubric that scores step efficiency and plan adherence, not just the end result.
  5. Get the GAIA slice. Create a free Hugging Face account if you don't have one, accept the gated-dataset terms, and pull a small slice (10–15 questions) of Level 1 (easiest) questions from the GAIA benchmark dataset. GAIA questions are real-world, tool-requiring questions across three difficulty levels.
  6. Run your agent against GAIA and score it. Run your agent on the GAIA slice, capture full trajectories, and reuse your DeepEval metrics from Steps 3–4 to score tool correctness, answer relevancy, and trajectory quality across the set. Compute a simple pass rate.
  7. Reflect on judge behavior. For at least 3 of your GAIA runs, manually read the trajectory and decide for yourself whether the agent succeeded. Compare your judgment to DeepEval's automated score. Note any disagreements and your hypothesis for why they occurred.
  8. (Optional, no extra credit required) If your results look solid, format and submit them to the public GAIA leaderboard for practice engaging with a real benchmark submission process.

Deliverables: Submit your DeepEval test file(s), a CSV or table of per-question scores (tool correctness, answer relevancy, trajectory score) for both your custom test set and the GAIA slice, and a one-page write-up covering your judge model choice, your custom rubric, and your Step 7 human-vs-judge comparison with at least one concrete disagreement analyzed.