Brian Teller: In this episode, I spoke with Saiyam Pathak about what it actually means to evolve a Kubernetes developer platform into an AI factory. We went beyond the models and focused on the infrastructure underneath them, how Kubernetes allocates GPUs, why assigning an entire GPU to a workload can create massive underutilization, and how technologies like MIG, MPS, time slicing, and HAMI make fractional GPU sharing possible. Saiyam also explained why Dynamic Resource Allocation matters, how it allows the scheduler to reason about GPUs as first-class devices rather than simple integers, and why isolation, workload-aware scheduling, observability, and token usage all become more complicated when multiple teams are running inference and training workloads on the same infrastructure. We also discussed where projects like HAMI, Volcano, Kueue, vCluster, and OpenTelemetry fit into the emerging cloud-native AI stack, and what platform engineers should start learning now. Listen to the full episode to understand how Kubernetes is becoming the foundation for shared AI infrastructure, and what still needs to mature before that experience becomes truly seamless. One of my favorite talks from KubeCon India wasn't about another AI model. It was about infrastructure. Saiyam, you argued that we've moved from building developer platforms to building AI factories. That's a pretty big shift. What does that mean?
Saiyam Pathak: So for a few years now, we have seen a new term that got emerged, right? The platform engineering, people building these platforms, the internal developer platforms, and then we started building these golden paths for internal developer platform, which is great. But now what has happened is we came into the AI wave. And people started talking about GPUs, like everything about GPUs. So now we have a few different teams and they want to use these GPUs without stepping on each other. And that's more of an evolution to what platform engineering was and people are building these AI platforms or AI factories. So the one thing that I want people to take away is that GPU is the new resource like it's not a new resource but it's the resource that people have started respecting for the AI or agentic AI workloads and Kubernetes has been handing these out as single digits, whole numbers. So this is where the whole concept of how we build the entire platform with keeping that every team now needs to run these agentic workflows and your same platform needs to be AI enabled. And for that, a lot of work you need to do on how you share those GPUs in an isolated way with maximum utilization to the teams.
Brian Teller: If someone missed your keynote, what's one idea that you'd hope they'd walk away with?
Saiyam Pathak: So I think one of the key things is that I focused on a few major things out of that. The main thing is how you can utilize the GPU in the most efficient way possible. So there are this like NVIDIA architectures, obviously the NVIDIA GPUs and there are different sharing patterns and different sharing techniques. So you have time slicing, you have MPS, you have MIGs and then how you in Kubernetes perspective, so those are like you can do on the hardware. Like MIG, you do on the hardware level and then another technique uses MPS server on the hardware itself. And then time slicing uses the software multiplexing. But when it comes to Kubernetes layer, which projects you do in conjunction with Kubernetes so that you can actually do the fractional GPUs was something that I wanted to showcase and where we are with the maturity. So one key takeaway was obviously one of the star projects of KubeCon India HAMI, which is now a CNCF incubating project. So I think that is something that people should.
Brian Teller: Okay, so what's fundamentally different than when I'm building a developer platform versus this AI platform? What do I have to worry about? What are the different trade-offs? How am I thinking differently between those two differences?
Saiyam Pathak: So people, especially talking from the enterprise perspective, people are building these AI factories. Now, these AI factories are something that you buy hardware from and people want to create that software stack and build the developer platform to share that stack across their teams for doing the AI workloads, whether it's inferencing, whether it's training. Although when you see the CNCF reports, they do mention that most of the work that is being done is on the inferencing and the fine-tuning side of things rather than training the model from scratch because now we already have these LLMs, we already have the open-source models which are out there, which are very capable enough of helping you out with assisting you out with coding or whatever that is or building the RAG pipelines or howsoever you want to take away. The fundamental difference is basically the resources have swapped like Kubernetes which has been natively built for CPUs is now handling or now expected to handle the GPUs which is expensive you have to buy those racks and you still want the same Kubernetes experience the same Kubernetes API underneath because the developers still do not want to see underneath. And the same concept of the developer platforms as well. Like developers do not care what is underneath. So they also want the same experience for the AI workflows. So that's the fundamental change, like getting the hardware, getting Kubernetes to understand that hardware, connect to the fleet of GPUs, and how when someone creates or uses it for a model for inferencing, then how it can be shared efficiently, utilized efficiently is something that a developer platform owner or the platform engineer or AI platform engineer or whosoever you want to call that will be building.
Brian Teller: So what do you think this conversation is happening now rather than just like a few years ago? What has changed recently that has caused that shift?
Saiyam Pathak: So obviously the scarcity of GPUs and that was one problem and then Kubernetes obviously started at DRA level and they started at a level where you can actually share, like you can actually attach a GPU. So, nvidia.com/gpu: 1 is something when you specify and then how the NVIDIA controller came up and started to make this available, make the GPU visible to the Kubernetes cluster. So it does all the device plugin things, it does the NVIDIA Container Toolkit, runs the DaemonSet and make sure everything is configured. So that was happening. That time, the sharing was not that much of the talk. And then NVIDIA open-sourced the project, which is their almost their Run:ai kind of thing, which is called KAI Scheduler, which is also, I think, a CNCF project. And then HAMI came in at the same time. And they already are now in incubating phase. And then if you see in Kubernetes 1.34, I believe so, DRA going GA. So these things started happening actually at that time when the core work was going on and since now that work is actually being used by the organizations in production so it is being talked more about like these things are possible people are building like these AI factories or neoclouds they are building these AI factories and they are building these inferencing engines or managed inferencing services so they do care about the GPU sharing, that's why there is a lot of talk about that. So inferencing has gone up. That's why the discussions about AI inference, like how GPUs will be used with the AI inferencing and how we can efficiently utilize and share the GPUs, which is why every conversation, like these conversations are happening at a very big scale across the industry right now.
Brian Teller: that makes sense. And I think like AI infrastructure has definitely become like an industry buzzword, but your keynote wasn't really about models as much as it was about the actual GPU becoming shared infrastructure. And it sounds like Kubernetes wasn't doing enough on its own. What's the biggest misconception that people have about running AI workloads on Kubernetes?
Saiyam Pathak: So I think what I believe is that people think the hard part is the model, like picking the model, serving the model and things like that. But I think the harder part is the GPU underneath. So like I mentioned before, when you do this nvidia.com/gpu: 1, and most people read this as okay, this is a GPU that gets attached, I use it. But this actually means that this particular GPU is held by that particular workload or a pod manifest or YAML the application and nobody will be able to use it even if you are using only 10% of that. So when we talk about inferencing, which has been the talk of the town, that is the real unsolved problem, like not unsolved, I would say, but kind of a misconception that model is fine, but GPU sitting there as a resource is what people miss to understand more, like, because that's the core that will be doing the inferencing and all the AI things.
Brian Teller: Okay, and in your keynote, you used Mumbai train station as an analogy for GPU utilization. Can you explain that to me? Like, how is that a useful comparison?
Saiyam Pathak: So I was going through like what I should present in a way that matches to the story that people are already living. And I saw a few things. And I think the rail infrastructure, the railways, the trains are the best maximum utilized infrastructure. So I told my keynote, like, if you give one compartment to one person, that can map to one GPU, nvidia.com/gpu: 1, and that can only you know at one point in time maximum 50 people can travel but train is actually making sure that more thousands and thousands of people are going from one place to another place and that's only because they share the infrastructure in a way that they take one place they sit over there and then it goes from one place to another and they utilize one compartment fully so that's a fully utilized infrastructure so I think that after the talk also many people came like oh wow this analogy really hits how utilization works and how the utilization of the GPU should be happening and besides this obviously I had the dabbawalas which are also one key theme and the kaali peeli taxis so kaali peeli taxis was for like you raise the hand and basically whenever you need a GPU you tell and the GPUs come to you And then whenever you don't need it goes back into the fleet and someone else can use it. So that was the taxi concept. And Dabbawalas are a tiffin system. So basically it has labeling, the proper labeling. So that was a case study done by Harvard as well, that how much efficient that whole system is. If you have the labels proper on a particular box, it reaches like which station, which building, which floor, that's it. And it reaches smoothly thousands and thousands with precision. Similar to that, how DRA then maps it, the same concept, and you treat the GPUs as the first-class citizens, and you have all the proper labeling, and the scheduler understands where it has to place, how it has to place.
Brian Teller: So you mentioned DRA. It's been getting a lot of attention recently, too. Why does it matter?
Saiyam Pathak: So DRA is Dynamic Resource Allocation. I think I always try to make people understand with the concept of persistent volumes. So when persistent volumes came in, it makes people understand that volumes are a first-class citizen. Like StatefulSets are the first-class citizens. Like the stateful workloads are the first-class citizens in Kubernetes. So similar to that, it has the concept where you have, you know, the what you call DeviceClass. And you also have ResourceClaim. The scheduler can finally reason about the devices instead of just the integer like you know nvidia.com/gpu: 1. So core DRA, I think it went through a lot of iterations but as of 1.34 it is GA, but it's pretty it's getting a lot of attention it usually does not slice it so it works very well in conjunction with the HAMI project or any other projects which are there And that's why I compared it to dabbawalas because of the rich information that you add, the rich labeling that you add, the routing that you add using DRA. So DeviceClass, ResourceClaim, it is really good for making GPUs a first-class citizen in Kubernetes.
Brian Teller: Interesting. Okay, so with that said, conferences, talks in general, I guess, they make things look clean, but as we know, like in not everything's an evergreen environment. There's a lot of complexity in the real world. You've spent a lot of time helping people build Kubernetes platforms. Where do teams end up actually struggling once they start deploying AI workloads in the real world?
Saiyam Pathak: Like it's a whole stack. So right now, the neoclouds and the AI factories, they are trying to understand how to provide value to the customers. Because what is happening to be very honest is people are buying a lot of hardware. I mean, they have, people are buying the capacity and actually the consumers, the enterprises are actually leasing or renting out for you know multiple year contract already when the orders are placed out but the real issue is not renting out the GPUs the real issue is providing value on top of that so many of the AI cloud providers are now moving towards providing a managed Kubernetes-like experience for their clouds like these are not the traditional big three providers the newer neoclouds or AI factories that people call them so these cloud providers want to provide a managed Kubernetes-like experience to their customers that have built-in GPU support. And second thing is people are also building the smaller services like, you know, managed inference engines. Like you just pick a model and it gets deployed behind the scenes and it gives you a model endpoint that you can do direct inferencing use in your coding agents and whatever that is and you can directly call that. So you can have a Kimi K2 or you can have a GLM model, you can pick it and you will be charged per million tokens kind of thing. So again, both of the scenarios, the biggest thing is the GPU sharing. So if you talk about the RTX PRO 6000 Blackwell 96GB, it supports MIG. So that means it can be sliced into different, you know, partitions at hardware level. So some people are using mounting that to a VM one slice to a VM and then giving that out to a customer in a smaller price some people are giving the complete entire RTX 96 GB card and leave that to the customer to you know do the shared infrastructure like how to slice up the GPUs some then the enterprise rents out the complete box so which has all the eight cards So eight RTX cards are there in the RTX PRO 6000 Blackwell. So they kind of pick the entire one. And similar to that, obviously, you have different series of the NVIDIA boxes. So that's the main challenge that I see is how to provide value and all boils down in the end, how to efficiently give that to the teams. So, for example, if I am a provider, it's my duty that I want my racks, my GPU racks to be fully utilized. And second thing is I want it to give it at a cheaper cost. Third thing is how well if I share my GPUs and if I do not have MIG enabled, how I can share those GPUs in a way that it is fair for the end users or the teams and it is also isolated. And how do I give those to them? So all these are different challenges that I see when people are building these kind of platforms.
Brian Teller: Is there Other operational issues or bigger operational issues, or is it just the GPU utilization?
Saiyam Pathak: So I think operational stuff, again, isolation is something that people try with namespaces and teams and stuff. This is where in the demo, I used a product called open source tool called vCluster, which gives you the tenant Kubernetes clusters. So every tenant having their own set of Kubernetes clusters with its own isolation layer, much more, obviously isolated than a namespace and things like that. These are very important because when you are building your own managed Kubernetes-like experience you want Team A, Team B, both to have the clusters and both to think they are the owner of those particular clusters so this can have Argo CD version v1 or v2 whatever that is 1.x and this can have Argo CD version 1.y. And they both should not interfere. They both should not be able to see each other. And as a provider, I want my GPUs and my CPUs to be utilized to maximum capacity. And this is where vCluster helps you create those isolated tenant clusters, and then you attach the GPUs, and then they are completely isolated from each other.
Brian Teller: Okay, so considering the isolation, considering the GPU utilization, how mature do you think the CNCF is, the CNCF ecosystem is today for AI?
Saiyam Pathak: So CNCF ecosystem is maturing a lot. Recently, in the past release, 1.35, the alpha feature came out called the workload concept. So natively, the Kubernetes scheduler cannot schedule batch jobs. It was not meant for the training kind of jobs so there one of the key things was like if training job requires eight GPUs, it takes two and it still stays in pending two are held by it doesn't get six so even two are wasted and it's not getting six so things are get wasted people use Volcano and some of the other scheduling you know some of the other CNCF projects to make this happen but Kubernetes community itself is working on Workload API to treat workload as a concept rather than Pod as a concept which is very interesting it's going you know very in the right direction Then you have projects like KAI Scheduler and in one of my demos that I gave was HAMI. So HAMI helps you to, you know, do the sharing on the GPU with fractions. Like you can do 40%, 20% and you can mention that. And it does at, you know, scheduling level first and then the runtime level. It actually sets the enforces the memory limits on that. So that is where you get. And then you can do the projects in conjunction. Like Kueue is one of the projects for jobs. So overall the CNCF ecosystem and they also have something which is called Cloud Native AI Landscape. So there is the CNCF landscape and then there is this CNAI Landscape. So that is also maturing a lot in terms of observability, agentic AI observability, how to measure the tokens in the right way. Then there are these projects like kgateway, agentgateway. Now these projects help you add a boundary layer. For example, when you call the MCP server, anyone can call it. But right now, the problem is we don't know because there are so many people calling. There is Team A calling one MCP server, Team B, and there might be some agents calling the MCP server. Everybody is able to call them. So there is no middle layer in between that. So that's where these gateways kind of sit so that they understand that you can add authorization. You can add something which is called token budgets. So you can budget your tokens to minimize the impact of excessive use, excessive abuse of the tokens. So multiple things are happening. And that was the essence of the talk, that Kubernetes, with its friends, is able to actually figure out how to actually run these, how to be the base layer for running these AI factories.
Brian Teller: So, okay, we've talked about GPU utilization and the issues with orchestration. Well, I guess the issues with allocating those GPU resources? And you mentioned observability. What would we monitor when it comes to these GPU workloads? And how is that different than how we would maybe monitor like a pod in Kubernetes?
Saiyam Pathak: So the thing is that the way NVIDIA exposes the GPU metrics is not traditionally how you capture the CPU metrics. So there are different tools like the DCGM Exporter is there. And then you can capture a certain amount of metrics. But in the AI workload ecosystem, observability even becomes trickier because of the agents and how they're consuming because people are charging on the basis of tokens. So very interesting, like how you map that utilization, the metrics of consumption of the GPUs to the utilization of the tokens, and then how you calculate the tokens and define the budgeting on top of that. So It's not extremely straightforward like the regular observability tools, but if you see the project in OpenTelemetry, there is one certain component that is totally focusing on the LLM kind of observability, the agentic observability. Then there are other open-source tools as well that focus on the observability. Like I think CNAI Landscape would have the what do you say, observability tooling as well. I mean, they do have certain observability tools.
Brian Teller: Okay, so gosh, with everything you just mentioned with CNCF, AI moves so quickly and it's hard to chase every new announcement. And most of it probably won't matter. So if you were a platform engineer today, what AI infrastructure trend is worth investing time in? And what's mostly noise?
Saiyam Pathak: So I think as platform engineers, you need to think from the perspective, whatever company you're working today or where you will be working tomorrow, that organization is definitely already either in the discussions of building that AI platform or adding capabilities to the existing developer platforms to cater those AI workloads or building a new one. So in the end, you have to transition to that and you have to learn the concepts. And in those concepts, I never say you should, I mean, if you want, definitely you should go as deep as you want. But it's not necessary to learn about the machine learning models, like the machine learning algorithms and going through the crux of that. But at least when you do a request to an LLM and when you get a response in between what happens from the prefill to cache to decode, you should have that level of knowledge because when you create that infrastructure and you give those endpoints to the customers and then they start hitting it at scale like 50 teams or 100 teams how the KV cache fills up so you should be able to understand that how the latency works you should be able to understand that same model being utilized by 10 folks how it happens and what request when it happens if some of the teams are working on training some on inferencing how you divide the infrastructure so that they do not interfere because inferencing is a short-lived kind of thing obviously based on context it can or reasoning it can do even a bit more. But training is kind of very longer lived than the inferencing overall so how you divide that so these things are something that people should definitely learn about obviously with these things there is tooling so you have this concept wise you have MLOps you have LLMOps you have AIOps all these ops but you have been the sysadmin for 25 years you know the terminology and the you know The terminology keeps on changing and, you know, the DevOps platform engineering and all that these things keep on evolving but the crux and the core remains the same in the end as an engineer who wants to deploy an AI agent I should be able to do that in a seamless manner and get the infrastructure in a seamless manner from the entire platform in the organization so that was the case before in the platform engineering that is the case today just that it adds more GPUs and it adds the world of AI into it, which is a bit different world than people are used to.
Brian Teller: that's for sure. Okay, so you had mentioned a little bit about the CNCF project or CNCF ecosystem. Is there a project specifically that people should be watching for? Do you think that there's some projects that will become more important over the next year or so?
Saiyam Pathak: I think HAMI is definitely the one that is picking up a lot. I mean, right now, there are fewer contributors, very active project, and the maintainers are still looking out for more and more contributors. So it's a pretty active project and people should get more and more involved in that because it directly hits the pain point of GPU sharing in Kubernetes and it does it in a seamless way. So that is something people should definitely check that out. Overall, like Kueue, Volcano, Kubeflow and in the observability space obviously OpenTelemetry with the AI capabilities these projects will grow a lot and you will see more and more adoption for all of these projects.
Brian Teller: That makes sense and overall do you where do you see like Kubernetes succeeding or struggling as this AI infrastructure world evolves?
Saiyam Pathak: I think struggle is real right now because it natively you Kubernetes is maturing, no doubt about that. The community is very active. But it still needs the projects, which is also good. That's why Kubernetes was extensible and that's how it is. But the concept of controllers and the concept of how that orchestration works is something that people are interested the most about, whether it's Kubernetes or not Kubernetes. But I think since people have invested a lot in building platforms already on the basis of Kubernetes, so people will keep their investments in and add the AI layer on top of that. And I do think that Kubernetes will become more boring. By that, what I mean is it will just be there and you don't realize, just like the operating system is there and you don't realize, you don't, you know, Ubuntu is there, it's fine. So you don't realize that. So Kubernetes is the layer, base layer that will be there for AI factories, for these neoclouds. And it will definitely be the base layer. Although every release, we see great new features and the health of the project, specifically Kubernetes, you can see how many alpha features are coming every edition. It is very cool because alpha features are the new set of features. And you can see there are more than 10 to 15 alpha features that come every release, which shows how much enthusiasm is there in the community to cater to newer type of workloads and newer type of scenarios.
Brian Teller: so given, I guess, these alpha features and what we see coming. Is there anything that you would, like if we were having this conversation a year from now, what do you think we'd be talking about? What would be the focus? What's interesting coming up?
Saiyam Pathak: I think Kubernetes, like in my head, Kubernetes should be able to self-heal itself in a way that you ask an LLM to do. So I think, you know, baked in something inside the scheduler itself that it automatically picks up the right stuff without anything, without any additional intervention, Workload API graduating, matured a lot more. So these DRA concepts, Workload API concepts, and the scheduler, making sure that the GPUs are the first-class citizens and everything just works. And if there are any issues, then automatically things get fixed. So I think that should be the world that people are heading into. And people are already building a lot of cool things, obviously with AI, giving them the superpowers. So the development cycles have increased. So hopefully we'll see cool stuff coming out of Kubernetes as well.
Brian Teller: So was there any other, was there any announcements or anything from KubeCon India that didn't get enough attention that are worth checking out?
Saiyam Pathak: I think inferencing is the key. So that's what was the whole, you know, game at KubeCon that people should worry, not worry, sorry, people should care more about inferencing and the work is not done yet so there is a lot of work that needs to be done like while I was preparing for the demo on the DGX Spark HAMI didn't work like HAMI didn't have the native support for working on DGX Spark so I contributed two pull requests to the project itself and I contributed back and then it worked out this is how the community needs to understand that they need to use these projects, but definitely think from their perspective, what is working, what is not, and contribute back so that the ecosystem becomes mature enough and it kind of builds out that, these are the tools to pick and then create that platform.
Brian Teller: Any closing thoughts for the audience?
Saiyam Pathak: I mean, I'm happy that I'm alive in this era and I'm able to see what is happening. And we, I mean, it's just like all the developers first just got these superpowers and people just need to use them the right way to do the right things.
Brian Teller: For sure. Saiyam, thank you so much. Really appreciate your time.
Saiyam Pathak: Thank you. Thank you so much.
