🎓 Final Capstone Project (100 pts)

Your task: design — and partially build — a production inference-serving system for a GPU-backed AI workload of your choosing. This is your chance to demonstrate, in one integrated project, everything AIINFRA 200 covered.

Scenario

Pick (or invent) a realistic scenario that needs a served model in production. Examples: a customer-support chatbot backend, an image-classification API for a mobile app, a document-summarization service for an internal tool, a recommendation-scoring endpoint, or a batch + real-time hybrid inference pipeline. Any scenario is fine as long as it genuinely requires the skills below — run it by your instructor early in the week if you're unsure.

Requirements

Your submission must include all three of the following components:

1. Written design document (2,000–3,000 words, submitted as a PDF or shared doc link). It must explicitly map your design decisions back to the skills you built across Weeks 1–15. At minimum, address:
1. Problem framing — the workload, its latency/throughput/SLA requirements, and why it needs dedicated inference serving rather than ad-hoc scripting (Weeks 1–3).
2. Model serving layer — which serving runtime/framework you chose, how the model is packaged and containerized, and the API contract it exposes (Weeks 4–6).
3. GPU orchestration — how the workload is scheduled on Kubernetes: resource requests/limits, GPU device access, and your autoscaling approach under load (Weeks 7–9).
4. Performance and cost optimization — your approach to quantization, batching, KV-cache/memory management, and/or multi-tenant serving, with a justification of the tradeoffs you made (Weeks 10–12).
5. Operations — how you'd monitor this system in production (key metrics/alerts), what reliability patterns you'd apply (circuit breakers, graceful degradation, rollout strategy), and a rough capacity/cost estimate (Weeks 13–15).
6. Known limitations — an honest section on what you did NOT solve or build, and what you'd need to do to take this from prototype to real production.
7. Cite specific tools, commands, or configurations you actually used — this should read like documentation of a real system, not a marketing pitch.
8. Grounded in the free/low-cost toolchain used throughout this course (e.g., open-source model servers, Docker, a local or free-tier Kubernetes cluster such as Minikube/Kind, and open-weight models) — no requirement to spend money on cloud GPUs.
2. Architecture diagram (image, PDF, or diagramming-tool export — e.g., draw.io, Excalidraw, Mermaid, or hand-drawn and photographed is fine as long as it's legible). It must show:
• The full request path from client to served model and back.
• Where GPU orchestration happens (cluster/node/pod boundaries).
• At least one operational element (monitoring, autoscaler, load balancer, or fallback path).
3. Working prototype artifact — a partial, runnable implementation proving out the riskiest or most central part of your design. This does not need to be the full system. Acceptable examples:
• A containerized model server (Dockerfile + a serving script) that you can run and query locally, with a sample request/response captured in your submission.
• A Kubernetes manifest (Deployment/Service/HPA, etc.) for your inference workload, validated with kubectl apply --dry-run or run on a local cluster, with output/logs captured.
• A working batching, quantization, or load-testing script exercised against a real or stubbed model endpoint, with results captured.

Include the code/config files themselves plus a short README (in the same submission) explaining how to run it and what you observed.

Deliverables

• [ ] Design document (PDF or doc link), 2,000–3,000 words, addressing all 6 required sections above
• [ ] Architecture diagram (image or PDF), legible and labeled
• [ ] Prototype artifact: code/config files + README explaining what it does and how to run it
• [ ] Everything submitted together as a single online upload (zip or multi-file upload) or combined into one document with links, via Canvas

Grading

Graded against the Capstone / Project Rubric (100 pts total). This assignment maps to all five course learning outcomes (CLO1–CLO5) — it is your comprehensive demonstration of course mastery.

💡 Tip: Don't try to build the "whole system" for real. The point of the prototype is depth on one piece, not breadth across all of them — your design document carries the breadth.