Title: Lecture 9 — Two Roads to Scalable Model Serving: Ray Serve and KServe
Course: AIINFRA 200: Production Inference Serving & GPU Orchestration > Section: Week 09 — Ray Serve and KServe for Scalable Model Serving

📖 Lecture — Two Roads to Scalable Model Serving: Ray Serve and KServe

So far in this course you have deployed models with basic containers and orchestrated them with Kubernetes fundamentals. This week we look at two production-grade serving systems that solve the scaling problem in very different ways: Ray Serve and KServe. Understanding both — and knowing when to reach for each — is one of the most practical architecture decisions you will make as an ML infrastructure practitioner.

Ray Serve: a programmable serving library

Ray Serve is a scalable, framework-agnostic model serving library built on top of Ray. "Framework-agnostic" means it doesn't care whether your model is PyTorch, TensorFlow, or Scikit-Learn — or even whether it's a model at all. Ray Serve can just as easily serve arbitrary Python business logic (a validation step, a feature transform, a rules engine) alongside a model. Because deployments are defined directly in Python code, Ray Serve is especially well suited to composing complex, multi-model inference pipelines — for example, chaining a preprocessing step, an embedding model, a reranker, and a generation model — without ever leaving the Ray ecosystem or writing YAML. A common misconception is that Ray Serve can only run on Kubernetes. It cannot only run there — it runs on standalone Ray clusters, a single laptop, a VM, or on Kubernetes via the KubeRay operator. This flexibility is a real strength: you can prototype a serving pipeline on your laptop and later deploy the identical code to a Kubernetes cluster managed by KubeRay, with no rewrite.

Ray Serve LLM: purpose-built for generative workloads

Ray Serve LLM extends Ray Serve with OpenAI-API-compatible deployment, meaning existing clients written against the OpenAI SDK can point at your Ray Serve LLM endpoint with minimal changes. Critically, its autoscaler reacts to queue depth, not just CPU utilization — a much better signal for LLM workloads, where a handful of long-running generation requests can saturate a GPU while CPU usage looks unremarkable. New APIs in Ray Serve LLM support advanced distributed serving patterns built on vLLM, including prefill-decode disaggregation (splitting the compute-heavy prompt-processing phase from the memory-bound token-generation phase across different resources), data-parallel attention, and expert parallelism (for mixture-of-experts models). A recent HAProxy-based ingress upgrade to Ray Serve LLM delivers up to 5x higher throughput and 8x lower latency versus the prior proxy — a reminder that the ingress layer in front of your model matters as much as the model-serving logic itself. A second misconception worth correcting directly: students new to Ray Serve autoscaling often assume replica count reacts instantly to raw request counts, as if it were a simple thermostat. In reality, the Ray Serve autoscaler monitors queue sizes to decide how many replicas are needed, and then Ray has to provision the underlying actors (the worker processes that run your code) to match. That means scaling has multiple layers of latency: detecting a queue buildup, deciding to scale, and then waiting for new actors to start and load the model. Expect scale-up to take seconds to low minutes, not milliseconds.

KServe: Kubernetes-native serving

KServe is a CNCF incubating project that provides the InferenceService custom resource definition (CRD) on Kubernetes. Where Ray Serve is a library you import, KServe is a Kubernetes-native abstraction: you declare an InferenceService object and KServe's controllers handle autoscaling, networking, health checks, and server configuration for you. KServe supports both predictive and generative model serving, with runtime support for Triton, Hugging Face Server, and others. This is the root of the most common misconception this week: assuming Ray Serve and KServe are direct substitutes solving the same problem the same way. They are not. Ray Serve is a general-purpose programmable library that can run anywhere — laptop, VM, or Kubernetes via an operator — and it excels at multi-model composition in code. KServe is built specifically around Kubernetes primitives (CRDs, controllers, the Knative-style serverless model) and excels when your organization's serving strategy is Kubernetes-first and you want infrastructure-level abstraction rather than code-level composition. For teams that need to serve many small models that change frequently, KServe offers an optional add-on called ModelMesh, enabled by annotating an InferenceService with serving.kserve.io/deploymentMode: ModelMesh. ModelMesh is designed for high-density, frequently-changing multi-model serving and supports a wide range of runtimes: TensorFlow, PyTorch, ONNX, scikit-learn, XGBoost, LightGBM, and OpenVINO IR.

Choosing between them

Factor | Choose Ray Serve | Choose KServe
Primary need | An all-in-one platform spanning training, tuning, AND serving | Kubernetes-native model serving only
Pipeline complexity | Complex, custom multi-model pipelines composed in Python | Standard single-model or ModelMesh-managed multi-model serving
Infrastructure | Multi-cloud, hybrid, or non-Kubernetes environments | Kubernetes-committed environments
Team background | Python/ML engineers comfortable writing serving logic in code | Platform/SRE teams comfortable with CRDs and Kubernetes operators
Model turnover | Moderate | Very high (many small, frequently swapped models via ModelMesh)

The table above is a decision aid, not a rulebook — plenty of organizations use both, with KServe fronting stable production endpoints and Ray Serve powering experimental or pipeline-heavy workloads.

Bringing it together

The throughline for this week: Ray Serve gives you programmable flexibility that runs anywhere, while KServe gives you Kubernetes-native standardization. Neither is objectively "better" — the right choice depends on your team's existing skills, your infrastructure commitments, and how complex your inference pipelines need to be. You'll get hands-on with both in this week's lab so you can feel the difference in developer experience firsthand, not just read about it.