In AIINFRA 300 and 301 you learned to build LLM apps and RAG systems that work. This week we start asking whether they work affordably. GPU FinOps is the discipline of measuring, predicting, and optimizing what inference actually costs — and it starts with getting the units right.
Everything in GPU FinOps derives from two related formulas:
Notice that both formulas have the GPU's hourly rental rate in the numerator and a throughput measurement in the denominator. The hourly rate is fixed the moment you pick a GPU and a cloud. The throughput is not fixed at all — it depends entirely on how you serve the model. That is the single most important idea in this lecture: the GPU's price tag is a constant, but your cost per request or per token is a variable that you control through utilization.
A request is not a unit of work — tokens are. A support-bot request that generates a 40-token answer and a document-summarization request that consumes a 5,000-token prompt and produces a 2,000-token summary are wildly different amounts of GPU work, yet both are "one request." If you quote a client "\$0.002 per request" without specifying token volume, you have told them almost nothing. A workload that runs a 5,000-token prompt 1,000 times has done the same amount of inference work as a workload that runs 5 million one-token requests. The rule: normalize on cost per token first, then translate to cost per request using your actual, measured prompt/completion sizes. Skipping this step is how teams end up with cost estimates that are off by 10x or more once real traffic arrives.
Here is the number that should change how you think about GPU cost: the same H100 running the same model (Mixtral-8x7B) can cost roughly \$15 per million tokens at 1 request/second, or under \$1 per million tokens at 25 requests/second. That's better than a 15x swing — on identical hardware, running identical weights. Nothing about the GPU changed. What changed is how densely the GPU's compute was packed with work. This happens because LLM inference is memory-bandwidth-bound, not compute-bound, for most of the decode phase. A GPU serving one request at a time spends most of its cycles waiting to move weights through memory for a single stream of tokens — the compute units sit mostly idle. Continuous (dynamic) batching lets the server interleave many requests' token-generation steps together, so each pass through memory produces tokens for dozens of requests instead of one. The hourly rental cost doesn't change; the tokens-per-hour denominator explodes, so cost per token collapses.
A common and expensive mistake is treating right-sizing as "pick the GPU class that guarantees nothing ever bottlenecks." In practice, right-sizing means matching GPU tier and count to measured load, not assumed peak load. A useful diagnostic: if SM (streaming multiprocessor) utilization stays below roughly 50% and VRAM usage stays below roughly 40% under real traffic, the workload is over-tiered — you are paying H100 prices for a workload an L4 or A10G could serve. Combined with continuous batching, right-sizing is typically the highest-ROI optimization available, often cutting self-hosted inference cost 40–60% with no change to the model or its outputs.
| Lever | What it changes | Typical impact |
|---|---|---|
| Right-sizing (matching GPU tier to load) | Fixed \$/hour cost | Can cut spend 40–60% by itself |
| Continuous/dynamic batching | Requests processed per memory pass | Can raise throughput 10–25x over naive single-stream serving |
| Utilization (req/sec against a given GPU) | Tokens/hour denominator | Drives 15x+ swings in cost per token on identical hardware |
| GPU vendor/market choice | \$/hour numerator | Typically 2–4x spread across providers (see pricing table below) |
Batching is not free. Larger batches raise throughput and lower cost per token, but they also increase the time any individual request waits to be scheduled and to receive its tokens — per-request latency rises with batch size. This means you cannot optimize cost in isolation. You must first fix a latency SLA (for example, "95th-percentile time-to-first-token under 800ms"), because that SLA is what caps how aggressively you can batch. Optimize cost before fixing the SLA and you will either violate your latency promise to users, or leave money on the table by batching too conservatively. This week's lab has you find this "knee of the curve" empirically.
Published on-demand rates vary substantially by cloud and by how much you're willing to commit:
| Provider | H100 On-Demand Rate | Notes |
|---|---|---|
| AWS | \~\$3.90/hr | Standard on-demand |
| Azure | \~\$6.98/hr | Standard on-demand |
| GCP (A3-High) | \~\$3.00/hr | As low as \~\$2.25/hr on spot |
| Lambda Labs | \~\$1.85–1.89/hr | Reserved pricing |
| Hyperbolic | \~\$1.49/hr | Specialized/GPU-marketplace cloud |
The spread between the cheapest and most expensive listed rate here is over 4x for the same GPU class — before you've even touched utilization. This is why FinOps work always starts with "what am I actually paying per hour," not an assumed list price.
Given all of the above, when does self-hosting actually make financial sense versus paying a hosted API per token? Worked example: at roughly 50 million tokens/day, calling the GPT-4o-mini API costs approximately \$2,250/month, while self-hosting an equivalent-capability model on 4× A10G GPUs costs approximately \$5,175/month — the API wins decisively. The crossover point, where self-hosting starts winning on pure infrastructure cost, is roughly 11 billion tokens/month (about 500 million tokens/day). Below that volume, the fixed cost of GPU capacity (which you pay whether it's busy or idle) outweighs the savings from not paying a per-token markup. Above it, owning the hardware and driving high utilization starts to beat the API's margin.
By the end of this week's lab, you will have measured — not looked up — your own throughput and latency curve, and you'll be able to defend a cost-per-1,000-requests number with real numbers behind it.