Title: Lecture 14 — Choosing a Deployment Model, and Watching the Bill
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 14 — Cloud, On-Prem & Hybrid: Deployment Models and Cost Awareness

📖 Lecture — Choosing a Deployment Model, and Watching the Bill

For most of this course you've thought about AI infrastructure in the abstract: compute, storage, models, pipelines. This week we ask the practical question every team eventually faces: where does that infrastructure actually run, and what does it cost? There are three answers — public cloud, on-premises, or a hybrid of both — and the whole skill is matching the model to the workload rather than defaulting to whatever is trendy. We close with cost awareness (FinOps), because the deployment decision is, at bottom, a cost-and-constraints decision.

A note on scope: this is a literacy-level comparison of deployment models, not a deep cloud build. Hands-on cloud platform work, identity/IAM, and containerized/Kubernetes deployment are covered in depth in AIINFRA 101 (Applied Infrastructure & Containerization); deep security is AIINFRA 302.

(a) Cloud: renting compute, pay-as-you-go

The cloud is where most beginners and small teams get their infrastructure. You rent compute, storage, and networking from a hyperscaler and pay by the hour, the API call, or the token. Each of the big three sells a flagship managed AI/ML platform so you don't have to hand-manage servers: AWS → SageMaker AI, Google Cloud → Vertex AI, Microsoft Azure → Azure AI Foundry (which also fronts OpenAI models inside Azure's compliance boundary). They exist for one reason: provisioning and scaling your own servers is slow and error-prone, so a managed platform takes on that operational burden — for a price.

A key distinction that applies far beyond AI is managed services vs. Infrastructure-as-a-Service (IaaS):

 | Managed services (SageMaker, Vertex AI, Azure AI Foundry) | Infrastructure-as-a-Service (EC2, Compute Engine, Azure VMs)
You control | Your data, model config, and code | The full OS, installed software, drivers
Provider handles | Provisioning, scaling, patching, much of the ML lifecycle | Only the hardware and virtualization layer
Billing | Usage-based (per call, per training hour, per token) | Per hour/second the VM runs
Best for | Teams that want to move fast without ops staff | Teams that need full control of the software stack

Neither is "better" — they sit on a control-vs-convenience spectrum. In practice, organizations rarely pick a cloud on a neutral feature comparison; they pick the one that matches what they already use for data and identity. A team living in BigQuery gravitates to Vertex AI; a Microsoft 365/Azure shop defaults to Azure AI Foundry. The right first question isn't "which is fastest/cheapest?" but "what does this organization already use?"

The "free" trap. All three offer genuine free ways to learn (AWS SageMaker Studio Lab needs no credit card; Google's "Always Free" e2-micro; Azure's \$200/30-day credit). But free tier does not mean free forever — every offer is bounded by a monthly cap, a time-limited trial, or a fixed list of resources, and once you exceed it, usage silently converts to billed usage.

(b) On-premises: owning your compute

On-premises (on-prem) means you own and operate the physical compute — a server, a workstation, or a home "lab" — instead of renting it. The trade-off mirrors the cloud's: on-prem swaps a large upfront capital cost and ongoing operational responsibility for long-term control and predictable costs. It's a spectrum, not one thing: a used RTX 3090 (24 GB VRAM) homelab build for ~\$500–600 is genuinely enough to run and fine-tune many open-source models, while enterprise-purpose-built servers (Dell PowerEdge XE, Lenovo ThinkSystem SR675i) run tens of thousands of dollars and up.

Owning a GPU is not the same as having a working AI machine: it needs the correct NVIDIA driver for your GPU/OS plus the CUDA Toolkit installed correctly — the most common cause of a "why isn't my GPU detected?" problem. (Driver/CUDA setup on real hardware is a hands-on topic; AIINFRA 101 goes deeper.)

On-prem tends to win when the workload is sustained and high-utilization (heavy use can break even against cloud rental in as little as 4–12 months), when data residency/privacy requires data never leave the building, or when a team wants predictable, fixed costs. Two misconceptions to correct: buying hardware is not a one-time cost — electricity, cooling, networking, and driver/firmware maintenance are real recurring operational costs, which is why total cost of ownership (TCO), not sticker price, is the right lens. And on-prem does not require an enterprise budget — but even a few-hundred-dollar machine carries those recurring costs.

(c) Hybrid: combining both

Most organizations don't pick one side — they land on hybrid cloud architecture: on-premises or private infrastructure combined with public cloud, each piece running wherever it makes the most sense. Typically, steady-state workloads run on owned hardware where utilization is high and ownership wins, while burst capacity and unpredictable demand go to the cloud, where flexibility matters more than unit cost. (Don't confuse hybrid with multi-cloud, which is using two public clouds side by side; a company can be either, both, or neither.)

Three forces push work toward the local side:

- Latency. A factory robot avoiding a collision or a camera flagging an event must decide in milliseconds; a round-trip to a distant cloud region is too slow, so inference happens near the sensor.
- Data transfer cost and volume. Sensors can generate terabytes a day; shipping all of it to the cloud is slow and expensive.
- Regulatory compliance. Some data legally cannot leave a building, region, or country.

Meanwhile, training large models at scale is exactly what the cloud is built for — on-demand GPU clusters you spin up for a few days and release. This tension is the entire reason hybrid exists.

Data gravity (coined by Dave McCrory, 2010): as a dataset grows, it becomes cheaper and faster to bring compute to the data than to move the data. Copying 500 GB is trivial; moving 500 TB generated continuously by factory sensors is not. This is why training and inference increasingly happen near where the data already lives.

Common hybrid patterns:

Pattern | How it works | Typical use case
Split-tier | Real-time processing runs on-prem/edge; aggregated analytics and reporting run in the cloud | Retail systems, manufacturing dashboards
Edge inference + cloud retraining | Model trained centrally in the cloud; a compact version runs inference on edge devices; only filtered data returns to retrain later | Cameras, autonomous equipment, IoT
Disaster recovery / burst | Primary workload runs on-prem; cloud capacity is reserved for failover or demand spikes | Seasonal retail traffic, business continuity

A useful nuance: "edge computing" almost never means training on the edge device — training stays centralized (it needs large data and heavy compute); what moves to the edge is a smaller, already-trained model doing inference only. Finally, data sovereignty (data is subject to the laws of the nation where it's collected/stored) and the stricter data residency (data must physically stay within a boundary) can be the deciding factor: if patient records legally can't leave the country, inference and storage must run on infrastructure inside that jurisdiction. Vendors now sell hybrid as a product (AWS Outposts, Azure Arc, Google Distributed Cloud): the same cloud APIs, but the hardware sits in your data center.

(d) Cost awareness: FinOps basics

The deployment decision is a cost decision, so you need the vocabulary to see costs coming. FinOps is not just a dashboard or a monthly report — the FinOps Foundation defines it as an operational framework and cultural practice that gets engineering, finance, and business into the same conversation to trade off speed, cost, and quality. Maturity is a Crawl–Walk–Run journey: Inform (visibility — tag resources, build dashboards, see who spends what) → Optimize (act — right-size, choose better pricing, cut waste) → Operate (make it routine — budgets, alerts, regular reviews).

What drives AI cost. GPUs are usually the biggest line item, and the hourly rate is only part of the story: a newer chip may cost more per hour but finish faster, lowering effective cost; utilization is the lever people forget — a GPU idle 70% of the time is still billed 100% of the time; large models are often bottlenecked by memory bandwidth, not raw compute; and training (long-running, batch, interruption-tolerant if checkpointed) differs sharply from inference (latency-sensitive, bursty). Beyond compute, watch the surprises that hit teams moving from a free-tier experiment to a real project: data egress fees (moving data out often costs more than moving it in), storage retrieval fees on cheap "cold" tiers, and idle managed endpoints that keep billing while unused.

Pricing models. Cloud providers sell compute several ways for different usage patterns:

Pricing model | Typical discount vs. on-demand | Best fit | Key risk
On-demand | 0% (baseline) | Unpredictable or short-term needs | Most expensive per hour
Spot | ~60–90% off | Fault-tolerant, checkpointed batch training | Can be reclaimed with ~2 minutes' notice
Reserved / Savings Plans | Moderate, steady discount | Predictable, steady baseline load | Requires committing to usage in advance

The biggest savings come from combining: Reserved/Savings Plans for steady baseline load, Spot layered on top for elastic, interruptible batch work. Right-sizing — matching instance size to actual observed utilization instead of guessing — is the other high-leverage move, and free tools like AWS Compute Optimizer read your utilization history and recommend it for you.

A simple cost-comparison mindset. For an LLM product, ask not "how much per month?" but "how much does one request cost?": cost = (input tokens × input price) + (output tokens × output price). Output tokens typically run 2–5× more than input tokens (each is generated by a full sequential forward pass), so tight prompts and prompt caching (reusing repeated context — up to ~90% savings) pay off. And the recurring myth — "the cloud is always cheaper than owning hardware" — is false in both directions: a cluster at 20% utilization belongs in the cloud, but at 50%+ sustained utilization with heavy steady volume, self-hosting can be cheaper. The right answer depends on your utilization pattern, not a blanket rule — which is exactly what this week's lab makes you calculate.

Next week (Week 15, the final) turns to reproducibility, testing, and responsible AI.