Bart Farrell: Sai, good to have you back on KubeFM. For people who don't know who you are, can you just let us know about what your role is and where you're working?
Sai Vennam: thanks for having me back, Bart. My name is Sai Vennam. I'm a Principal Solutions Architect at AWS. I love working with customers on driving through challenges by leveraging Kubernetes at scale, especially today with agentic workloads being the focus for so many customers. So it's exciting to work towards solutions to these challenges. And also, I personally love running events and workshops to help customers learn with actual hands on keyboards as well.
Bart Farrell: More aggressive bin packing reduces node count, but it also increases contention and failure impact. How far should teams push consolidation before the reliability trade-off stops being worth the savings?
Sai Vennam: you know, this is actually a really interesting question and I think calls out an interesting misconception that a lot of people have. The way Karpenter scales is on resource requirements: requests, not actual usage. So what you actually want to do is push your workloads until, you know, there's a disruption that it can't handle, right? So essentially, push until you hit the disruption that each workload can tolerate, not necessarily like a node count target, like this many workloads per node. So to start, you rightsize all the requests that your workloads have. I know that's a tall order, but start with rightsizing. Set your memory requests equal to the limits first. And then you leverage things like Karpenter's disruption budgets. That lets you be maybe more aggressive off hours, more careful at the peak. And for long-term training jobs like agentic jobs, you can be more hands-off. And of course, I think I just started with the obvious answer here is Karpenter. I think the industry as a whole agrees that Karpenter is the right solution for autoscaling workloads and consolidation. Starting with that assumption there.
Bart Farrell: And when we ask people about their favorite tools, Karpenter comes up time and time again. So I'm pretty sure the community agrees with that wholeheartedly. Now, GPU utilization is especially easy to misread because an expensive accelerator can be busy without being efficiently used. Which metrics actually tell you whether a Kubernetes GPU workload is cost efficient?
Sai Vennam: you know, the GPU utilization metric can be misleading because it just tells you that the GPU ran at least one instruction. So a single tiny kernel read can read at 100%. So this is why sometimes when you're running workloads, you'll just see it spiking up and down. So there's the NVIDIA DCGM exporter that exposes signals that I trust a lot more here. Things like power draw, there's tensor core activity. And you also want to tie it to business outcomes. So things like cost per token, or cost per training step. And on Amazon EKS, there's the CloudWatch Observability add-on. This actually deploys NVIDIA's DCGM, I think I have those letters right, exporter for you. So those metrics are there out of the box.
Bart Farrell: Last but not least, Kubernetes can have workload autoscaling and node autoscaling making decisions at the same time. How do you prevent those control loops from fighting each other and creating cost or stability problems?
Sai Vennam: So in general, Karpenter and some of these other autoscalers were designed to work with each other, but you're right, they can compete. So in short, it would be to give each autoscaler one job and one signal. Obviously, there's more signals at play here, but that's maybe one way to think about it is maybe one job per scaler. So you can use something like HPA, Horizontal Pod Autoscaler, or KEDA to scale pods on real demand, like queue depth. And then use Karpenter only for nodes when pods can't be scheduled, which is kind of the default way that Karpenter operates. Or remove nodes that no one's actually using through bin packing. Then you kind of align their timing. So there's Karpenter's consolidation delay. And that should be at least as long as HPA's scaled down window, essentially preventing race conditions. I know a lot of these things seem a little specific, but there are well-established patterns for how to do this more precisely. And I think AWS, we've actually published a few blogs on this as well. So be sure to check those out.
Bart Farrell: Excellent. Well, as usual, Sai, great catching up. Thanks so much for sharing your knowledge with the community and look forward to speaking to you soon. Take care.
Sai Vennam: Thanks for having me, Bart.
