This exam covers Weeks 1–7. The answer key below is hidden from students in the published Canvas quiz and is included here only for instructor reference.
Scenario: You are the infrastructure engineer at a mid-size company that wants to deploy an open-weight 13B-parameter chat model for internal employee use (roughly 200 employees, bursty usage concentrated in business hours). Leadership wants: (1) the model containerized and deployable to a Kubernetes cluster, (2) GPU resources used efficiently since the team only has a small shared GPU pool, (3) the ability to autoscale down to zero overnight to save cost, and (4) basic observability so the team can see request volume and latency. Using what you've learned in Weeks 1–7 (containerization, Kubernetes fundamentals, cloud fundamentals, GPU resource management, and your first production serving deployments), describe your end-to-end approach: how you would containerize and deploy the model, how you would handle GPU scheduling/sharing, how you would configure autoscaling (including scale-to-zero), and what observability you would add. Be specific about the tools/concepts you would use at each step. → Model answer: Containerize the model server (e.g., a vLLM or TGI serving container) with the model weights either baked in or pulled from object storage/model registry at startup, and define resource requests/limits including GPU count via a Kubernetes device plugin (e.g., NVIDIA GPU Operator) so the scheduler places pods only on GPU-equipped nodes. For efficient sharing across a small GPU pool, consider time-slicing or MIG partitioning if the GPUs support it, and set appropriate nodeSelector/tolerations so GPU pods land correctly. Deploy via a Kubernetes Deployment plus a Service (and Ingress if external access is needed), and use a Horizontal Pod Autoscaler or KEDA-based scaler driven by request-queue-depth or custom metrics rather than CPU alone, since GPU inference load doesn't track CPU usage well; KEDA (or a serverless GPU platform) enables true scale-to-zero overnight, spinning a pod back up on the first incoming request (accepting a cold-start/model-load latency hit). For observability, instrument the serving container with Prometheus-compatible metrics (request count, latency percentiles, queue depth, GPU utilization) and visualize with Grafana, plus basic logging/tracing so the team can see request volume and correlate slow periods with GPU saturation. A strong answer names concrete tools (Kubernetes, GPU Operator/MIG, KEDA/HPA, Prometheus/Grafana) and explicitly ties each design choice back to one of the four stated requirements.
→ b
→ False. A Deployment manages pod replicas and self-healing, but routing/load-balancing traffic across those replicas requires a Service (ClusterIP/NodePort/LoadBalancer) in front of them.
→ Any two of: reduced operational burden (control plane managed for you), built-in upgrades/patching, tighter integration with the cloud provider's networking/IAM/storage, and easier access to managed GPU node pools.
→ b
→ False. GPU inference workloads are typically GPU- and queue-bound, not CPU-bound, so CPU-based HPA often fails to scale appropriately; custom metrics (GPU utilization, queue depth, request latency) are preferred.
→ A container image is the static, immutable packaged artifact (filesystem layers + metadata); a container is a running instance of that image with its own writable layer, process(es), and runtime state.
→ b
→ True.
→ Service (internal ClusterIP or NodePort) and Ingress (or a LoadBalancer Service) for external access.
→ b
→ True. Health checks alone don't reveal degraded performance (e.g., high tail latency or GPU saturation) that a healthy-looking pod can still exhibit.
→ RPS describes capacity/volume handled but says nothing about the latency or tail-latency experienced by individual users, so a high-RPS system could still violate a latency SLO for real users.