Bart Farrell: Welcome to KubeFM. First, things first, Andrés, for people who don't know you, who are you, what's your role, and where do you work?
Andrés Valero: Thank you for having me today, Bart. My name is Andrés Valero. I work at SUSE as a technology advocate, a fancy title for saying I do something between marketing and technical stuff. I sit between both. I've been working with Kubernetes for the last nine years or so. It's kind of a long time and I'm happy to be here.
Bart Farrell: Now, we're going to be speaking about right-sizing cost optimization. Andrés, where do most teams get Kubernetes right-sizing wrong when they move raw CPU and memory usage to an actual resource decision?
Andrés Valero: In my experience, it's just trusting what comes from Prometheus or whatever tool they use, because in there, the spikes are hidden or softened. That's a tricky one. It's a good source of information, but you cannot trust it as 100% accurate. You need to be careful. It depends on what kind of applications. For instance, if you're using Java, an old version of Java is not container-aware. You may have a container that has a limit of two gigabytes of RAM, but Java is not aware of it. That container can keep growing and growing until it gets killed by Kubernetes, by the Linux kernel, because it's exceeding its memory. That's a tricky one. You need to make very aggressive cuts on CPU because that may drive HPA a little bit crazy. If you go the other way, I'm going to cut in the RAM. If you cut on CPU, your application may be slower, but won't be killed. Whereas if you go short on memory, your application will be killed and you will lose service. Right-sizing is very tricky. And at the end, when you are trying to cut costs, you need to remove nodes or hosts from Kubernetes. That's where the actual savings will come. You need to focus on, we make sure that memory is good enough so it never will be killed by the Linux kernel because of the memory limit. We configure the horizontal autoscaler in a way that it scales with sense. Set up the limits and the requests right. Otherwise, you will find yourself in a situation where either you will have many more pods than you need, or you will end up with a pod that is not able to carry the load of your application. That's where I would go with this.
Bart Farrell: And Andrés, how do you right-size aggressively enough to reduce waste without creating HPA surprises, CPU throttling, or OOM risk?
Andrés Valero: First, we need to make sure that we don't fall in the HPA trap. We need to set up the request and the limit consciously. We can let the application grow without spinning containers, like pods in this case, like crazy. But we could also see how to do this. For instance, you can get the VPA in recommendation mode. It gives you information. Not changing anything, just collecting that information. You observe your workloads because what you see in your Prometheus or whatever tool you're using is the past, not the future. You need to look for the sweet spot. You can be brave with CPU, although it is tricky, as we said, because HPA can go crazy. And you have to be very careful with memory. If you want to start being aggressive, you can do safe tiers, a 20% cut maximum. See what happens. You can do that in a non-production environment. See what happens. Although you need the data for the production environment, but you can start testing there, doing some load tests on your own, making sure that whatever happens there, it is strong enough to simulate a real-world scenario and not go 60% configuration that will end up having a lot of pods in any given moment. You can make configurations, making sure that VPA and HPA complement each other. Since I think it was Kubernetes 1.35, we can have the redeployment of the limits and requests of a pod without killing the pod. You can give a little bit extra limit on VPA at the same time you are keeping under control. HPA. They can complement each other without going crazy. Obviously, you need to be careful because depending on the Kubernetes platform you're using, on how configured you may be even and having extra nodes instead of less nodes because you are spawning pods and pods and pods. Be careful, do your testing, do your homework, make sure that you have some kind of load test tool that allows you to test what is the behavior before going in production, and start with something safe, something small, not go crazy on it.
Bart Farrell: What does a production-grade feedback loop from Prometheus data to an approved GitOps resource change actually look like?
Andrés Valero: Let's get our data on Prometheus. Let's get a month of history. We can get SLO metrics. We can have VPA in recommendation mode and collect the data coming from that. We start getting more real-world data that is what we want. We want to make sure that we collect this real-world data. Once we get that data, we can have different scenarios where we can put these changes on the limits or in the HPA on a Git repo. We have a tool, it can be Flux, Argo CD, whatever your GitOps tool of choice for doing this, and we make sure that all the changes are coming from a single source. We need to make sure that our data is right. Second, we have a single source of changes. We can enforce that the changes are only coming from whatever data source we decide can be our Git repo. We can use Kyverno or OPA. There are different tools. We can even use Kubewarden and there are different tools that we can use to enforce policies and make sure that changes are only good if they are coming from a certain place, a certain Git repo. And once we have that, we need these changes to be approved and reviewed by someone that knows the environment, and we can start rolling out from there. The idea is, we observe, we get solid data, we make a proposal, we have a single source of truth that will be our Git repo, we control that no one else is messing with that on the cluster directly, just coming from our Git repo. Then we start a feedback loop. You implement, you try, you test, see the results. This is working, this is not working. Then re-evaluate and re-implement if necessary. The idea is making sure that the drift is minimal and your workloads are secure. For instance, in this case, AI is everywhere now. For this, we could use AI to have an agent that observes, that modifies and adapts. That's something that is completely doable with today's tools. If you're brave enough or you have the resources, why not create an agent? Make sure that the agent is checking all this data because that's something that AI is going to do way better than us. AI is good with data, is good with changes that are really restricted. Don't let it go crazy. You observe this data, you ingest this data, you analyze the data, you see the source, you see what's happening on the cluster, and then you can add. But only change in this. You can have it limited.
Bart Farrell: And Andrés, what's next for you? What are you going to be working on or learning next?
Andrés Valero: It's almost event season for me. This time I'm not going to KubeCon North America. I would be already preparing our demos for Edge Computing and Telco, but in February we start with SUSECON, then Mobile World Congress, and then KubeCon, all in Barcelona. I'm going to be living in Barcelona for a month. That's next for me. Start preparing and renewing my CNCF certifications that are long overdue, to be honest.
Bart Farrell: If people want to get in touch with you, what's the best way to do that?
Andrés Valero: Go to LinkedIn. It's avaleror. It's my name and my last name with an R at the end. You can go to my webpage. It's andresvalero.tech. There is an email there you can reach me. Available to help if I can answer questions or if you want me to talk at an event. I'm happy to do so as long as the schedule works for us. That's all. I'm pretty easy to find there are not a lot of Andrés Valeros around. I can say it was very easy for me, so thanks so much for sharing your knowledge with us today. I look forward to speaking to you again in the future and best of luck with all your preparations. Take care. Thank you. Take care. Bye.
