Title: Lecture 12 — Serverless Compute and Managed Kubernetes for ML Workloads
Course: AIINFRA 101: Applied Infrastructure & Containerization > Section: Week 12 — Managed & Serverless Kubernetes in the Cloud

📖 Lecture — Serverless Compute and Managed Kubernetes for ML Workloads

Earlier in this course you learned to run containers yourself — building images, starting them with Docker, and standing up your own Kubernetes clusters (Weeks 6–9) before moving into cloud foundations (Weeks 10–11). This week we look at two ways cloud providers take the host-management burden off your hands entirely: serverless functions and managed Kubernetes. Both promise "don't worry about the servers," but they deliver on that promise very differently, and knowing the difference is essential for choosing the right tool for an ML workload.

AWS Lambda: pay only for what runs

AWS Lambda is the canonical serverless compute service. You upload a function (or a container image), and AWS runs it in response to events — an HTTP request, a file upload, a queue message. You are billed per request and per millisecond of execution time, with zero charge when the function is idle. There is no server to provision, patch, or pay for while it sits waiting. Under the hood, every Lambda invocation runs inside an isolated Firecracker microVM, giving each execution strong security isolation without the overhead of a full VM. Each function has a hard 15-minute maximum execution time — a signal that Lambda is designed for short, bursty, event-driven work, not long-running services. Because AWS manages the fleet of microVMs behind the scenes, Lambda auto-scales from zero to thousands of concurrent executions with no capacity planning on your part.

Correcting the cold start myth

A cold start happens only when Lambda needs to create a brand-new execution environment — the first invocation, or a new one spun up to handle increased concurrency. It does not happen on every invocation; once a warm environment exists, subsequent requests routed to it skip initialization entirely. A related misconception: a scheduled "keep-warm" ping only keeps one instance warm — if ten concurrent requests arrive, nine still hit cold starts. Guaranteeing a true warm pool requires Provisioned Concurrency, a paid feature that partially undoes Lambda's "pay only for what you use" proposition, and is the right tool for latency-sensitive, predictable-traffic APIs.

Fargate: serverless containers

AWS Fargate extends the serverless idea to containers on ECS or EKS. Instead of provisioning EC2 instances to host your containers, you declare the CPU/memory your task needs, and Fargate provisions the underlying infrastructure automatically, including OS patching — a meaningful contrast with EC2-backed ECS/EKS, where your team still owns the node fleet.

Managed Kubernetes: what "managed" really covers

"Managed Kubernetes" (EKS, GKE, AKS) generally means the provider manages the control plane — API server, etcd, scheduler — not the whole system. Customers still handle node management, upgrade planning, IAM/RBAC, network policy, and cost governance, unless they've opted into a further-abstracted mode.

Offering | Control plane | Node/data plane management | Pricing model
EKS (standard) | AWS-managed | Customer manages EC2 worker nodes | \$0.10/hr control plane + EC2 node cost
EKS Auto Mode | AWS-managed | AWS automates provisioning/patching via Karpenter + Bottlerocket | Control plane fee + compute, 99.95% SLA
GKE Autopilot | Google-managed | Google fully manages the data plane too | Billed per pod resource requests
AKS | Azure-managed | Customer manages node pools (unless using Automatic mode) | Free control plane + node cost

EKS Auto Mode automates node provisioning and patching using Karpenter and Bottlerocket, backed by a 99.95% SLA. GKE Autopilot goes furthest in abstracting the data plane — Google fully manages node provisioning, sizing, and patching, and you're billed on the resources your pods actually request, not raw node-hours. A frequent misconception: pods don't scale automatically just because you're "in the cloud." Autoscaling requires an explicitly configured HorizontalPodAutoscaler (HPA) — nothing scales by default.

KServe: purpose-built model serving on Kubernetes

For ML workloads specifically, KServe (a CNCF Incubating project) provides an InferenceService CRD that wraps up GPU scheduling, NVIDIA Triton integration, dynamic request batching, autoscaling (including scale-to-zero), and built-in health checks. Instead of hand-rolling Deployments, Services, and HPAs for every model, you declare an InferenceService spec and KServe handles the primitives underneath.

Choosing a managed ML platform, briefly

Cloud providers also offer managed ML platforms — Amazon SageMaker AI, Vertex AI (rebranded the Gemini Enterprise Agent Platform in April 2026), and Azure Machine Learning — that sit on the same control-vs-convenience spectrum as the compute options above.

Platform | Philosophy | Best for
Amazon SageMaker AI | Granular control — you configure EC2 instance types, VPCs, security groups | Teams wanting fine-grained cost/performance tuning
Vertex AI / Gemini Enterprise | Abstracted infrastructure — gcloud ai custom-jobs create auto-packages your code into Docker and launches it | Teams wanting to move fast without infra expertise
Azure Machine Learning | Balanced, with AutoML and prompt flow for low-code LLM apps | Teams in the Microsoft ecosystem, or wanting AutoML

Two gotchas to remember: stopping a SageMaker notebook stops compute billing but not the attached EBS volume, which keeps billing per GB-month until deleted or resized; and Vertex AI/Azure ML process data within your specified region and don't use customer data to retrain foundation models by default, though data can still move between data centers within that broader region. As with compute, there's no single "best" managed ML platform — the choice depends on your team's expertise, control needs, and ecosystem investment.