Bart Farrell: First things first, who are you, what's your role, and where do you work?
Andrew Hillier: I'm Andrew Hillier. I'm the CTO and one of the co-founders of Kubex.
Bart Farrell: Now, when does right-sizing actually reduce the bill rather than just lowering pod requests without removing any nodes?
Andrew Hillier: We see this quite a bit. The requests are the things that cause the autoscalers to run as many nodes as they do. So as you increase your requests, they go up. I think a lot of times when you bring down the request, they don't actually bring down the nodes because sometimes you have to give it a bit of active help. And people have heard of things like bin packing, node consolidation. So if that doesn't happen, then the bill doesn't come down because you keep running on a lot of nodes. So we'll see a case, for example, where workloads expand out and you'll have, for argument's sake, 50 nodes running. And then the workloads go back down and all the nodes are running 60% utilized. And a lot of times nothing will happen. You're just still running on 100 nodes. Different technologies have different consolidation capabilities. Karpenter can do active consolidation if you enable it and tell it how to do it. OpenShift has consolidation thresholds. A lot of times we'll need to go in and tune those thresholds to say, if you're not using more than 70% of the capacity, start to actively bring down the nodes. So you can configure consolidation in various technologies. We also have an active descheduler that can go and actively drain nodes to get things off them. So again, sometimes environments behave nicely and come down the road; sometimes they need a bit of help. Again, bin packing is one of the main methods.
Bart Farrell: How should teams coordinate rightsizing with HPA and VPA so that they don't save per pod, but accidentally scale out more?
Andrew Hillier: This is a tricky area because a lot of vertical autoscaling technologies will conflict with the horizontal scaling technologies. And in HPA, the real problem there is that if HPA is scaling on a metric like CPU, saying if the CPU hits a certain percentage of request, run more. And so if you bring down that request, it'll start running more on its own. It'll just scale out. And we've seen that happen if you don't coordinate that properly. And the problem there becomes: a lot of times those replicas all use a fixed amount of memory. So if you start to scale those things out, even though they're not busy for CPU, they all use a fixed amount of memory and you start chewing up a lot of memory as you get this kind of unbounded scaling. So there are several ways to deal with that. Of course, you can not touch the metric that HPA is using to scale on, or you could be a little smarter about how you do it to make sure that you understand the scaling threshold and not size the container in a way that triggers the threshold inadvertently. So it is safe to resize them, but you have to understand what it's going to do: something sitting at 10% all the time and the scaling threshold is 90%. You have a lot of room to play with to downsize the request, but if it's close, it will cause more scaling. So the key is to just make sure that it's all coordinated when you do it. Because like you said, you can cause massive swings in the scaling if you start messing with it and the threshold. And so one way is at that level just to make sure you're coordinating the requests to not cause that to happen. The other one that we approach is we take an agentic approach to say, let's look at the entire HPA setup. Let's look at its historical metrics, its scaling behavior. Did it have that memory kills or throttling? Did it reach its upper bound or lower bound? And just holistically come up with more intelligent settings forward. And that might be: bring the min down a bit, bring the max up a bit, change the threshold to here, maybe also scale on memory. So with AI, it's really nice because you can use agents to say, let's look at all the settings and the history and actually set them all together because it's an art to tune HPA a long time. So that's the other approach. So instead of just sizing the requests, look at everything and come up with better settings overall. And the scaling threshold is a big part of that because we see them all over the place. Some people use 50, 80, 120. There are different ways to think about it and you get wildly different configurations out there. And again, it creates a lot of complexity.
Bart Farrell: And what evidence do you need beyond CPU and memory metrics before deciding a workload is genuinely safe to rightsize?
Andrew Hillier: Even within the CPU and memory, I think it can be very complicated. Memory itself: there are multiple metrics that you need to take into account. Is it working set, resident set, the total memory? Generally, the working set is one of the most important ones because you want to make sure you're giving it what it's actively using, but the buffering can help as well. So I think all those things need to be considered when even just sizing memory. In CPU, if somebody's obviously not using the CPU, you can bring it down. Limits have inadvertent effects on things if you bring them too low or too high. And the thing we see a lot now is things like JVMs: if you start resizing the CPU without looking at the garbage collector concurrency, you can mess them up. So we see a lot of times where a JVM has a lot of CPU, and you can't just bring it down because if you do, for example, it'll flip from ZGC to G1GC. It'll start using a different garbage collector because that's triggered by resource allocations and it's different in different versions of Java. So sometimes certain versions use the limit; certain versions use the host size. So sometimes you have to go a little level deeper into the workload to make sure that you understand what it's doing before you change the CPU because it can actually change, again in this case, garbage collector. Or Go workloads, a lot of them have these behaviors where you can't just take a surface level look at the CPU. So those are the ones that come to mind. Again, memory is complicated in itself. CPU, you gotta be careful. It's also important to look at the entire node because really, your request doesn't really do anything. It helps everybody else, all the other containers on the node. If your request being higher protects everybody else, it doesn't guarantee you get anything because if everybody else is busy, they're going to stomp on you. So you also have to look at the node level saturation, imbalances. If my nodes are running out of CPU because there's a whole lot of other workloads that are misconfigured, I have to be very careful. And so you have to look at a system and say, when I go after these things, am I able to go after the sizing or is the system messed up right now and I need to fix other workloads first? And then again, for certain workloads, a lot of times the more important ones, there are a lot of deeper considerations before you can just start resizing them. Again, the JVMs are a great example of it, which are everywhere and big resource users and a bit tricky to get right.
Bart Farrell: Excellent. Andrew, thank you so much for sharing your time and experience with us today. Look forward to speaking to you again soon. Take care.
Andrew Hillier: Thanks, Bart.
