Title: Lecture 13 — Cost per 1,000 Requests, Cost per Token, and the Utilization Lever
Course: AIINFRA 302: AI Security, Guardrails & Governance > Section: Week 13 — GPU FinOps Foundations: Cost per 1,000 Requests and Right-Sizing

📖 Lecture — Cost per 1,000 Requests, Cost per Token, and the Utilization Lever

In AIINFRA 300 and 301 you learned to build LLM apps and RAG systems that work. This week we start asking whether they work affordably. GPU FinOps is the discipline of measuring, predicting, and optimizing what inference actually costs — and it starts with getting the units right.

The two formulas that matter

Everything in GPU FinOps derives from two related formulas:

- Cost per 1,000 requests = (GPU \$/hour) / (requests/hour) × 1,000
- Cost per million tokens = (cluster \$/hour) / (tokens/sec × 3600) × 1,000,000

Notice that both formulas have the GPU's hourly rental rate in the numerator and a throughput measurement in the denominator. The hourly rate is fixed the moment you pick a GPU and a cloud. The throughput is not fixed at all — it depends entirely on how you serve the model. That is the single most important idea in this lecture: the GPU's price tag is a constant, but your cost per request or per token is a variable that you control through utilization.

Why "cost per request" alone is misleading

A request is not a unit of work — tokens are. A support-bot request that generates a 40-token answer and a document-summarization request that consumes a 5,000-token prompt and produces a 2,000-token summary are wildly different amounts of GPU work, yet both are "one request." If you quote a client "\$0.002 per request" without specifying token volume, you have told them almost nothing. A workload that runs a 5,000-token prompt 1,000 times has done the same amount of inference work as a workload that runs 5 million one-token requests. The rule: normalize on cost per token first, then translate to cost per request using your actual, measured prompt/completion sizes. Skipping this step is how teams end up with cost estimates that are off by 10x or more once real traffic arrives.

Utilization is the dominant lever

Here is the number that should change how you think about GPU cost: the same H100 running the same model (Mixtral-8x7B) can cost roughly \$15 per million tokens at 1 request/second, or under \$1 per million tokens at 25 requests/second. That's better than a 15x swing — on identical hardware, running identical weights. Nothing about the GPU changed. What changed is how densely the GPU's compute was packed with work. This happens because LLM inference is memory-bandwidth-bound, not compute-bound, for most of the decode phase. A GPU serving one request at a time spends most of its cycles waiting to move weights through memory for a single stream of tokens — the compute units sit mostly idle. Continuous (dynamic) batching lets the server interleave many requests' token-generation steps together, so each pass through memory produces tokens for dozens of requests instead of one. The hourly rental cost doesn't change; the tokens-per-hour denominator explodes, so cost per token collapses.

Right-sizing is not "buy the biggest GPU"

A common and expensive mistake is treating right-sizing as "pick the GPU class that guarantees nothing ever bottlenecks." In practice, right-sizing means matching GPU tier and count to measured load, not assumed peak load. A useful diagnostic: if SM (streaming multiprocessor) utilization stays below roughly 50% and VRAM usage stays below roughly 40% under real traffic, the workload is over-tiered — you are paying H100 prices for a workload an L4 or A10G could serve. Combined with continuous batching, right-sizing is typically the highest-ROI optimization available, often cutting self-hosted inference cost 40–60% with no change to the model or its outputs.

Lever | What it changes | Typical impact
Right-sizing (matching GPU tier to load) | Fixed \$/hour cost | Can cut spend 40–60% by itself
Continuous/dynamic batching | Requests processed per memory pass | Can raise throughput 10–25x over naive single-stream serving
Utilization (req/sec against a given GPU) | Tokens/hour denominator | Drives 15x+ swings in cost per token on identical hardware
GPU vendor/market choice | \$/hour numerator | Typically 2–4x spread across providers (see pricing table below)

The three-way tension: latency, throughput, cost

Batching is not free. Larger batches raise throughput and lower cost per token, but they also increase the time any individual request waits to be scheduled and to receive its tokens — per-request latency rises with batch size. This means you cannot optimize cost in isolation. You must first fix a latency SLA (for example, "95th-percentile time-to-first-token under 800ms"), because that SLA is what caps how aggressively you can batch. Optimize cost before fixing the SLA and you will either violate your latency promise to users, or leave money on the table by batching too conservatively. This week's lab has you find this "knee of the curve" empirically.

What GPUs actually cost right now

Published on-demand rates vary substantially by cloud and by how much you're willing to commit:

Provider | H100 On-Demand Rate | Notes
AWS | \~\$3.90/hr | Standard on-demand
Azure | \~\$6.98/hr | Standard on-demand
GCP (A3-High) | \~\$3.00/hr | As low as \~\$2.25/hr on spot
Lambda Labs | \~\$1.85–1.89/hr | Reserved pricing
Hyperbolic | \~\$1.49/hr | Specialized/GPU-marketplace cloud

The spread between the cheapest and most expensive listed rate here is over 4x for the same GPU class — before you've even touched utilization. This is why FinOps work always starts with "what am I actually paying per hour," not an assumed list price.

Build vs. buy: a concrete break-even

Given all of the above, when does self-hosting actually make financial sense versus paying a hosted API per token? Worked example: at roughly 50 million tokens/day, calling the GPT-4o-mini API costs approximately \$2,250/month, while self-hosting an equivalent-capability model on 4× A10G GPUs costs approximately \$5,175/month — the API wins decisively. The crossover point, where self-hosting starts winning on pure infrastructure cost, is roughly 11 billion tokens/month (about 500 million tokens/day). Below that volume, the fixed cost of GPU capacity (which you pay whether it's busy or idle) outweighs the savings from not paying a per-token markup. Above it, owning the hardware and driving high utilization starts to beat the API's margin.

Correcting the misconceptions

1. "Cost per 1,000 requests is a fixed, comparable number." It isn't — it's meaningless without token volume per request, which can vary by orders of magnitude. Always normalize on cost per token first.
2. "Self-hosted GPU cost per token is a constant I can look up." It is dominated by utilization and batching. The same GPU, same model, can vary 15x or more in cost per token depending purely on request rate.
3. "Right-sizing means picking the biggest GPU so nothing bottlenecks." It means matching GPU tier to actual measured load. Over-provisioning to an idle big GPU — running an H100 at 1 req/sec when an L4 with proper batching would do — is one of the most common and expensive mistakes in production LLM infrastructure.

By the end of this week's lab, you will have measured — not looked up — your own throughput and latency curve, and you'll be able to defend a cost-per-1,000-requests number with real numbers behind it.