If vLLM already solved LLM serving, why did SGLang appear?
Understanding Kubernetes for AI
Ten years ago, Kubernetes was a Google-incubated container orchestration project that most
enterprises hadn’t heard of. Today it’s the infrastructure layer underneath some of the most
demanding AI workloads in the world, and the teams that got there earliest have been quietly
building a head start that’s getting harder to close.
OpenAI has been on it since 2016, and their Head of Infrastructure, Christopher Berner, described why they’ve stayed on it, ‘Because Kubernetes provides a consistent API, we can move our research experiments very easily between clusters.’ That consistency is the same reason 82% of container users are now running Kubernetes in production and why the entire MLOps tooling ecosystem built itself around it. For enterprise teams making infrastructure decisions for AI, the tougher question is growingly about how to use Kubernetes well.
But using Kubernetes well for AI requires more than just pointing it at GPU workloads and
scaling up. The teams that have gotten the most out of it have made specific choices about how
they manage shared resources, handle bursty workloads, and set up the tooling around it – and
those choices matter a lot more than most teams expect going in.
Why AI Workloads Push Infrastructure to Its Limits
Running AI at scale puts specific demands on infrastructure that most orchestration tools
weren’t designed for. A few that matter most:
- GPUs are expensive and easy to underutilize without a scheduling layer that allocates them intelligently across competing workloads and teams.
- Distributed training requires parallelizing computation across dozens of nodes simultaneously, with those nodes communicating continuously throughout the job.
- Inference demand is unpredictable – a model update or a product launch can trigger spikes that statically provisioned infrastructure simply can’t absorb fast enough.
- Training jobs are stateful in a way web applications aren’t: a job that crashes at hour six needs to recover from where it left off, not start over.
- AI pipelines span multiple stages – ingestion, preprocessing, training, evaluation, serving. They need to work together as a system, not as disconnected processes managed separately.
These are the problems Kubernetes for AI addresses. The reason it addresses them better
than the alternatives is that the assumptions it was built around – ephemeral workloads, shared
scarce resources, frequent failure, and dynamic scaling map directly to what AI infrastructure
actually has to deal with.
Simplify AI Workload Orchestration
Deploy and scale AI workloads with managed Kubernetes built for performance and flexibility.
Where Kubernetes for AI Makes the Biggest Difference
The generic case for Kubernetes is well-documented. What’s less talked about is the specific
set of capabilities that make it the right fit for AI workloads in particular. And why teams that
have been running it for AI long enough tend to point to the same handful of things as the ones
that actually moved the needle.
GPU scheduling that works across shared infrastructure:
Without a proper scheduling layer, GPU access across teams becomes an ad hoc process.
Whoever asks first gets the resources. Kubernetes solves this through namespaces, resource
quotas, and priority classes that enforce access rules consistently across every team and
workload. NVIDIA’s GPU Operator extends this further, handling detection, scheduling, and
isolation automatically. A significant reason why NVIDIA made Kubernetes the default
assumed infrastructure layer in its enterprise AI reference architectures.
Auto-scaling that handles bursty workloads without manual intervention:
Training runs need burst compute. Inference spikes without warning. Kubernetes handles both
through Horizontal Pod Autoscaling, Vertical Pod Autoscaling, and KEDA. Scaling workloads up
when demand hits and back down when it drops – without anyone provisioning capacity
manually in either direction.
The same operational model across every environment:
AI workloads don’t live in one place – training in the cloud, sensitive workloads on-prem,
inference increasingly at the edge. Kubernetes gives teams the same APIs, deployment
patterns, and policies across all of these. That’s exactly what Berner was describing at OpenAI, and it’s the capability enterprise teams with multi-environment AI infrastructure tend to value
most once they’ve been running this way long enough to have a real opinion.
Self-healing for long-running jobs:
A training run that crashes at hour six is expensive. Kubernetes restarts failed containers,
reschedules workloads onto healthy nodes, and maintains desired state without manual
intervention – which changes what on-call looks like significantly for teams running jobs that take
hours or days to complete.
Reproducible experiments through declarative configuration:
Kubernetes’ declarative config model means the exact state of a workload – compute resources,
environment variables, container images, dependencies can be version-controlled and
recreated exactly, in any environment, by any team member. The days of ‘it worked on Tuesday
and nobody knows why’ get a lot shorter.
The MLOps Ecosystem Completed the Picture
Kubernetes for AI didn’t win on platform capabilities alone. The MLOps tooling ecosystem built
itself around Kubernetes, and that ecosystem is now as much a part of the value proposition as
the platform itself.
- Kubeflow for distributed training management and pipeline orchestration
- MLflow for experiment tracking and model versioning
- Ray and KubeRay for large-scale distributed computation across heterogeneous infrastructure
- KServe and Seldon Core for Kubernetes-native model serving with autoscaling built in
- Argo Workflows for end-to-end pipeline orchestration
- JupyterHub for shared notebook environments
None of this happened randomly – these tools were built where the workloads were running, and
Kubernetes’ extensibility made it possible to build AI-specific tooling as first-class citizens of the
platform rather than workarounds bolted on after the fact.
The dynamic that emerged is self-reinforcing. Kubernetes attracted the tooling, the tooling made
Kubernetes more capable for AI, which brought more teams, and drove more tooling
investment. According to Spectro Cloud, 90% of organizations expect their AI workloads on
Kubernetes to grow over the next twelve months. At this point, building an AI infrastructure stack
without Kubernetes means building outside the ecosystem that the rest of the industry is actively
investing in.
What Kubernetes Still Doesn’t Fix
The Linux Foundation found that 47% of organizations cite cultural change as the top barrier to
cloud native adoption, ahead of complexity, cost, and security. The teams scaling AI well on
Kubernetes have adopted GitOps, built shared platform infrastructure, and invested properly in observability. The teams struggling have deployed Kubernetes without changing how they work
and found the complexity worse, not better.
A few things worth being clear Kubernetes doesn’t solve on its own:
- The data problem is separate. Kubernetes manages and scales workloads, but
fragmented or ungoverned data produces bad AI outputs regardless of how well the
orchestration layer runs. We covered what that actually costs enterprise AI teams in our
recent blog. - Observability doesn’t come configured out of the box. OpenTelemetry, Prometheus, and
Grafana need proper setup to give real visibility into what’s happening inside AI pipelines
at runtime. - GPU cost management still requires active attention. Utilization patterns in AI workloads
can mask waste in ways that are hard to see without purpose-built monitoring, even with
Kubernetes scheduling doing its job. - Security across shared multi-tenant AI infrastructure is a dedicated operational
discipline, not something that defaults to safe
The Compute Underneath Your Kubernetes Stack
Kubernetes decides how workloads get scheduled and managed. It doesn’t determine what the
compute underneath looks like or what it costs, and for enterprise teams running serious AI,
that’s where the next meaningful decision sits.
Neysa Velocis provides GPU infrastructure built to sit underneath Kubernetes-based AI
workloads, across NVIDIA L4, L40S, H100 SXM, H100 NVL, H200 SXM, and AMD MI300X.
For teams that have already built a Kubernetes-native AI stack, Velocis fits into existing
orchestration patterns without requiring a different operational model on top of it. If you’re
thinking through what the compute layer needs to look like as your Kubernetes for AI
infrastructure scales, the Velocis team is worth talking to.
