Bart Farrell: First things first, who are you, what's your role, and where do you work?
William Rizzo: My name is William Rizzo. I'm a global field CTO at Mirantis, based in the Netherlands, and I work a bit everywhere with customers globally.
Bart Farrell: William, what three emerging Kubernetes tools are you keeping an eye on?
William Rizzo: The first one would be Kueue. I think with AI workloads, we are starting to realize that scheduling is only part of the problem. Kubernetes is very good at answering the question where can I run this workload? But when you have multiple teams competing for a limited number of GPUs, there is another question that becomes more important: should this workload be allowed to run now or later on? For example, should an experimental training job get access to capacity if a production inference workload may need that capacity five minutes from now or one hour from now? Can one team temporarily borrow unused quota from another one? Or what happens when the original owner needs that quota back? Those are not really scheduling questions, they are admission and resource governance questions. And I think Kueue becomes very interesting because it doesn't sit beside the scheduler, it sits on top of the scheduler before something is scheduled. That would be one. The other one, which we're developing some stuff around, is Hami. That's how I pronounce it, H-A-M-I. It's also a CNCF project, and it's interesting because it's resolving the problem of GPU sharing. Right now we can split a GPU into a fixed number of partitions. I'm talking about MIG GPUs. Giving every user an entire GPU makes absolutely no sense in the enterprise. With Hami, things become interesting because you can create this heterogeneous pool of GPUs and slice percentages of this pool and assign it to workloads. I find this interesting because it takes a different approach to sharing the resources. Another one is Kagent. This is one that is also interesting to me because it gets closer to the AI augmented operations part of the discussion. When we say AI augmented operations, we have this fear. Now we have AI running kubectl commands against our production and who knows what's about to happen. But I think that Kagent and agents can be useful in gathering evidence and helping the engineers form hypotheses on incidents. And they can do it very fast. What I'm less interested in is simply giving an LLM cluster admin to run kubectl commands. For me, the value is reducing the time we spend collecting and correlating information, not removing the engineer from what actually happened. These are my interesting technologies at the moment.
Bart Farrell: What tends to break first when AI workloads move from a few experiments to shared production Kubernetes infrastructure? Which problems are actually Kubernetes problems versus problems that teams are simply pushing onto Kubernetes?
William Rizzo: What I've seen breaking so far is not Kubernetes at all. It is the assumption we made when we designed the platform. If you have two or three engineers experimenting with models, you can get away with a huge amount of manual intervention. That's no problem. That's the experimentation/development phase: Someone allocates a GPU, someone knows which model is running. If something goes wrong, everybody knows who owns it. This works surprisingly well in an environment where it's not at large scale, it's a small environment. But then you move to 10 teams, multiple models, shared GPUs, infrastructure, development, and so on. Suddenly, the whole operating model that should have been used during your testing phase, it just falls apart. You start seeing resource contention, capacity problems, noisy neighbors, queuing and ownership questions. I think it is important to at least separate three different kinds of problems. The first category is generally a Kubernetes problem. For example, the workload cannot be scheduled, the device is not exposed correctly, the topology requirements cannot be addressed, or the drivers that are exposed to the containers are not the correct ones from the GPU device, or something like that. These are platform problems that can be easily solved like we normally did, even with telco, when we're dealing with hardware assigned to workloads, we did the same things. But then there is a second category, which is application or workload problems. Imagine I have a vLLM inference service. The pod is running. Kubernetes says that it's healthy. The GPU exists. From Kubernetes' point of view, everything is spick and span. Everything is fine. But the user suddenly is experiencing a terrible latency. The UX, of course, is horrible. Maybe the KV cache is under pressure. KV cache is key-value cache. Maybe concurrency is increased, or maybe the requests are queuing inside the inference engine. Maybe the model is using more memory than before. Kubernetes cannot understand by itself all of this. For Kubernetes, the pod is ready. And we have the saying in platform engineering that a pod being ready is not application health, it's not a healthy application. You have to evaluate the health of the application. We have to do the same thing with the AI workloads. I think the most important lesson with AI workloads is that a running pod doesn't necessarily mean a healthy application. Then there is a third category, which is really an organizational policy. Imagine a research team is running a long experiment and consuming most of the GPU capacity. Then a production inference service suddenly needs scale. There are no GPUs left. So is Kubernetes broken? Is the cluster having issues? No, there is just not enough capacity left to schedule. So Kubernetes may be doing exactly what you told it to. What you failed to define was the policy. Does production have priority? Can development be preempted? All those activities that we do already when we do capacity management on our platform, we should apply to AI. However, with AI workloads, we are not looking at the same metrics and the same characteristics of the workload as we would with more traditional workloads.
Bart Farrell: GPU capacity is expensive and finite. How should platform teams think about the trade-off between utilization, workload isolation, developer self-service, and predictable performance when multiple AI teams are competing for the same infrastructure?
William Rizzo: That's a good one. The first I would challenge is the assumption that maximum GPU utilization should always be the goal. Of course, GPUs are expensive, and nobody wants to have very expensive hardware sitting idle. We want to make money. But let's say I manage to achieve 95% GPU utilization. As a result, my production inference service starts missing its latency target. Have I really optimized? My infrastructure may look fantastic, but I'm providing a worse service than I should. I think a better objective is maximum useful work within the performance and isolation requirements of the workload. Useful is in quotes. It may mean very different things. For an inference service, it could be the requests served, the latency, the time to first token, or cost per request. For training, it could be the throughput or the time to completion. GPU utilization is important, but it's not the only signal that we should look at. The other point is that different AI workloads need different treatment. Take development environments, for example. Someone is experimenting with a model. They might use the GPU for 10 seconds, then spend five minutes changing some code. Giving that person the entire GPU exclusively may make no economic sense. Sharing is probably perfectly reasonable there. Take a look at Hami. Love that project. Now compare that with production inference where we have a latency SLO that we must respect with the business. For that workload, I may deliberately accept lowering the utilization if it gives me predictable performance. Distributed training is another category. Finding eight available GPUs in my environment is not enough. Those GPUs might need the correct PCIe topology, NUMA placement, NVLink or NVSwitch connectivity. There are factors that are not as straightforward as they are with classic workloads. Eight GPUs in the wrong place may behave differently from eight GPUs with the correct topology. I don't think we can apply a single allocation policy to every AI workload or AI infrastructure. The same applies to the sharing technology. Time slicing with NVIDIA MIG can increase GPU utilization. But there was a talk from NVIDIA at KubeCon Paris, where they showed that the incorrect configuration of MIG can lead to something like a maximum of 35 to 37% utilization of each partition. Even then, even when you're sharing, you might not optimally utilize your GPU partitions. Take a look at Hami. There is a lot of automation there.
Bart Farrell: Very good. Shout out to Hami. Shout out to Kueue and Kagent. Solid principles. As AI workloads become another production dependency, what operational decisions should teams be willing to automate or augment with AI? And which ones still require an engineer to understand the evidence and make the call?
William Rizzo: I don't think we should introduce AI into places where deterministic automation already works very well. Kubernetes is already an extremely sophisticated automation system. Controllers are constantly reconciling state. If containers die, Kubernetes can restart them. If a replica disappears, Kubernetes can replace it. If the desired state says five replicas, Kubernetes doesn't need an LLM to understand what five means. For me, AI becomes interesting when the evidence is incomplete, distributed across different systems or requires a little more interpretation. Imagine that an inference service suddenly becomes slow. An AI-assisted system could look at Kubernetes events from etcd, GPU telemetry, vLLM metrics, and maybe discover that the latency increased over a short period of time when a model was deployed. At the same time, the KV cache pressure went up or the request queue depth increased. That is useful. Now the engineer has a plausible hypothesis and, more importantly, the evidence behind that hypothesis. AI assistants and agents can help with that, and I think that is critical. Operational AI should not simply give us answers that sound convincing. It needs to show us why it reached that conclusion. We are naturally lazy and we often accept very quick answers that sound very good. AI is particularly good at doing this. I think AI is particularly good at things like diagnostic collections, correlation, incident summarization, configuration drift, and other drift-related tasks. We should definitely incorporate it into our operations.
Bart Farrell: Big shout out to the Hami Project. Can't wait for that swag to drop.
William Rizzo: Absolutely. I would love some swag.
Bart Farrell: I would too. I hope to get some at KubeCon. William, what's next for you?
William Rizzo: What's next for me? To be honest, there are a few talks coming up. I have to be at KCD UK and KCD Porto. Shout out to those folks. They are doing an amazing job. Then I'll be at the Open Source in Finance Forum in New York and KubeCon Salt Lake City. Looking forward to KubeCon as well. Lots of customer work, which I have to do while I'm preparing all these talks.
Bart Farrell: A lot of juggling and a fair amount of travel. I hope you stay hydrated. Thanks so much for sharing your time and your knowledge with us today, and I look forward to seeing you at KubeCon Salt Lake City very soon. Take care, William.
William Rizzo: Absolutely. Thank you, Bart.
Bart Farrell: Let's go, Project Hami. Cheers.
