Title: Discussion 15 — Why Are GPUs So Often Underutilized?
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 15 — Observability, Cost Analysis, and Right-Sizing

💬 Discussion 15 — Why Are GPUs So Often Underutilized? (20 pts)

Industry surveys show that over 75% of organizations run their GPUs below 70% utilization even at peak load, and only about 7% ever exceed 85% utilization — despite GPUs being the single most expensive resource in most AI infrastructure budgets. In a post of at least 200 words, answer:

- Based on what you've learned this week (and across the course), what do you think are the primary barriers preventing organizations from achieving higher GPU utilization? Consider technical causes (batching strategy, request patterns, model/hardware mismatch), organizational causes (team incentives, procurement lead times, fear of over-committing capacity), and observability causes (teams not measuring the right things in the first place).
- Which single operational or architectural change do you believe would have the highest impact on utilization for a typical mid-sized inference deployment, and why? Be specific — reference a concept from this week (e.g., dynamic batching, quantization, right-sizing GPU class) or an earlier week.

Then respond to at least two classmates, engaging with a barrier or solution they raised that you did not mention in your own post.