Title: Midterm Exam
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 08 — Serving Engine Benchmarking Lab

📝 Midterm Exam (100 pts)

This exam covers Weeks 1–7. The answer key below is hidden from students in the published Canvas quiz and is included here only for instructor reference.

Part A — Applied Scenario (40 pts)

Scenario: You are the infrastructure engineer at a mid-size company that wants to deploy an open-weight 13B-parameter chat model for internal employee use (roughly 200 employees, bursty usage concentrated in business hours). Leadership wants: (1) the model containerized and deployable to a Kubernetes cluster, (2) GPU resources used efficiently since the team only has a small shared GPU pool, (3) the ability to autoscale down to zero overnight to save cost, and (4) basic observability so the team can see request volume and latency. Using what you've learned in Weeks 1–7 (containerization, Kubernetes fundamentals, cloud fundamentals, GPU resource management, and your first production serving deployments), describe your end-to-end approach: how you would containerize and deploy the model, how you would handle GPU scheduling/sharing, how you would configure autoscaling (including scale-to-zero), and what observability you would add. Be specific about the tools/concepts you would use at each step. → Model answer: Containerize the model server (e.g., a vLLM or TGI serving container) with the model weights either baked in or pulled from object storage/model registry at startup, and define resource requests/limits including GPU count via a Kubernetes device plugin (e.g., NVIDIA GPU Operator) so the scheduler places pods only on GPU-equipped nodes. For efficient sharing across a small GPU pool, consider time-slicing or MIG partitioning if the GPUs support it, and set appropriate nodeSelector/tolerations so GPU pods land correctly. Deploy via a Kubernetes Deployment plus a Service (and Ingress if external access is needed), and use a Horizontal Pod Autoscaler or KEDA-based scaler driven by request-queue-depth or custom metrics rather than CPU alone, since GPU inference load doesn't track CPU usage well; KEDA (or a serverless GPU platform) enables true scale-to-zero overnight, spinning a pod back up on the first incoming request (accepting a cold-start/model-load latency hit). For observability, instrument the serving container with Prometheus-compatible metrics (request count, latency percentiles, queue depth, GPU utilization) and visualize with Grafana, plus basic logging/tracing so the team can see request volume and correlate slow periods with GPU saturation. A strong answer names concrete tools (Kubernetes, GPU Operator/MIG, KEDA/HPA, Prometheus/Grafana) and explicitly ties each design choice back to one of the four stated requirements.

Part B — Short Answer (60 pts, 5 pts each)

1. (MC) Which of the following best describes the primary purpose of containerizing a model server? (a) It makes the model more accurate (b) It packages the model, dependencies, and runtime into a portable, reproducible unit (c) It automatically adds GPU acceleration (d) It replaces the need for a Kubernetes cluster

→ b

1. (TF) True or False: A Kubernetes Deployment guarantees that traffic is load-balanced across pod replicas without needing a separate Service object.

→ False. A Deployment manages pod replicas and self-healing, but routing/load-balancing traffic across those replicas requires a Service (ClusterIP/NodePort/LoadBalancer) in front of them.

1. (Short answer) Name two reasons a team might choose a managed Kubernetes offering (e.g., GKE, EKS, AKS) over self-managed Kubernetes.

→ Any two of: reduced operational burden (control plane managed for you), built-in upgrades/patching, tighter integration with the cloud provider's networking/IAM/storage, and easier access to managed GPU node pools.

1. (MC) In GPU resource management, what problem does NVIDIA MIG (Multi-Instance GPU) primarily solve? (a) Increasing total GPU memory (b) Partitioning a single physical GPU into isolated smaller GPU instances for multi-tenant sharing (c) Automatically converting CPU workloads to GPU workloads (d) Compressing model weights

→ b

1. (TF) True or False: Horizontal Pod Autoscaling based purely on CPU utilization is generally a reliable signal for scaling GPU-bound inference workloads.

→ False. GPU inference workloads are typically GPU- and queue-bound, not CPU-bound, so CPU-based HPA often fails to scale appropriately; custom metrics (GPU utilization, queue depth, request latency) are preferred.

1. (Short answer) What is the difference between a container image and a running container?

→ A container image is the static, immutable packaged artifact (filesystem layers + metadata); a container is a running instance of that image with its own writable layer, process(es), and runtime state.

1. (MC) Which cloud fundamentals concept most directly determines your monthly bill for a GPU instance left running idle overnight? (a) IAM roles (b) On-demand compute pricing / lack of autoscale-to-zero (c) DNS configuration (d) Container registry choice

→ b

1. (TF) True or False: Scale-to-zero is generally appropriate for latency-insensitive or bursty workloads, but introduces a cold-start latency tradeoff.

→ True.

1. (Short answer) List two Kubernetes primitives (objects) you would use to expose a model server both inside and outside the cluster.

→ Service (internal ClusterIP or NodePort) and Ingress (or a LoadBalancer Service) for external access.

1. (MC) What is the main advantage of pulling model weights from object storage at container startup rather than baking them into the image? (a) Faster inference (b) Smaller/more reusable images and easier model version updates without rebuilding (c) Better GPU utilization (d) Required for Kubernetes compatibility

→ b

1. (TF) True or False: Observability for a production inference deployment should include request latency percentiles and GPU utilization, not just uptime/health checks.

→ True. Health checks alone don't reveal degraded performance (e.g., high tail latency or GPU saturation) that a healthy-looking pod can still exhibit.

1. (Short answer) In one sentence, explain why "requests per second" alone is an incomplete metric for evaluating a serving deployment's readiness for production traffic.

→ RPS describes capacity/volume handled but says nothing about the latency or tail-latency experienced by individual users, so a high-RPS system could still violate a latency SLO for real users.