Title: Discussion 8 — GPU-Bound or I/O-Bound? Reading the Signals
Course: AIINFRA 101: Applied Infrastructure & Containerization > Section: Week 08 — Deploying an AI Inference App to Kubernetes

💬 Discussion 8 — GPU-Bound or I/O-Bound? Reading the Signals (20 pts)

Imagine you're operating the inference service you deployed in Lab 8, but now it's serving real production traffic. What metrics would you monitor to figure out whether your workload is GPU-bound (the GPU itself is the bottleneck) versus I/O-bound (something like network latency, disk reads, or data preprocessing is the bottleneck) — and how would that distinction change your scaling strategy? In your initial post (150+ words), name at least three specific metrics you'd watch (e.g., GPU utilization, request queue depth, request latency, CPU utilization, network throughput), explain what a GPU-bound signature versus an I/O-bound signature would look like in those metrics, and describe how you'd scale differently in each case (for example: adding more GPU replicas vs. optimizing a data-loading step or adding CPU-only preprocessing Pods). Remember from the lecture that high GPU utilization alone doesn't prove your workload is healthy — build that caveat into your answer. Then respond to at least two classmates, comparing your monitoring approach to theirs and pushing on any place where "high GPU usage" might be getting misread as "efficient inference."