The Karpenter Effect: Redefining Kubernetes Operations

Bart: In this episode of KubeFM, we're joined once again by Tanat, an SRE at Adevinta, returning for a second appearance on the show. In our previous conversation, we touched on Kubernetes upgrades and the challenges of managing large-scale clusters. This time, we go deeper, exploring how Tanat's team redefined their Kubernetes operations with AWS Karpenter across more than 30 clusters and over 2,500 nodes.

We'll unpack what it takes to operate at this scale, from decoupling control and data plane upgrades to managing pod and node disruption budgets, to implementing flexible instance selection that saved the company over €30,000 a month. Tanat also explains how his team uses Kyverno to automate policies, streamline developer workflows, and keep platform operations predictable, even under heavy load.

We'll talk about the lessons learned from integrating observability, balancing cost and performance, and tuning Karpenter for real-world efficiency. If you're running Kubernetes at scale, or preparing to, this conversation is full of practical insights, architecture lessons, and reliability wins from someone who's been in the trenches.

This episode of KubeFM is sponsored by LearnKube. LearnKube has been providing Kubernetes trainings all over the world since 2017. Trainings are given in-person, online, to groups, to individuals, and the courses are instructor-led. They are 60% practical and 40% theoretical, and students have access to the course materials for the rest of their lives. For more information, check out LearnKube.com.

Now, let's get into the episode. Welcome back to KubeFM. Very nice to have you with us. What are three emerging Kubernetes tools that you're keeping an eye on?

Tanat: Sure. The first one would be Kubernetes itself. The latest version 1.34 has just been released with cool features to check out. For example, Kustomize is now in alpha, which should solve indentation and core issues in manifests. There are also interesting additions like pod-level resource requests and limits, and mutating admission policy is worth looking at.

The second tool I would recommend is KCP. I found this project from an episode of KubeFM. It's used to build a logical Kubernetes workspace for multi-tenancy use cases on top of a single cluster, allowing each workspace to have its own set of Custom Resource Definitions (CRDs), which solves many problems.

The last tool I'd pick is DeepSeek. It's not exactly related to Kubernetes, but I really like its concept: using a large language model to analyze your code and create an interactive wiki page that you can chat with and ask questions about your source code in a more interactive way.

Just in case someone didn't listen to the previous episode, I'll quickly introduce myself. I'm Tanat, a Site Reliability Engineer (SRE) at Adevinta, based in Barcelona. As part of the runtime team, I build and maintain an internal Kubernetes runtime platform called Ship, which hosts several digital marketplaces across Europe for Adevinta.

Bart: Tanat, previously you joined us in the other episode in Barcelona, but today you are located in a different place. Where is that?

Tanat: I'm in my hometown of Bangkok, Thailand.

Bart: Very good. Now, in terms of your background and experience, how did you get into Cloud Native? What were you doing before Cloud Native?

Tanat: The story is that my former company was starting a digital transformation project to digitize the system. I think I got lucky to be part of it because it exposed me to things like containerization, twelve-factor apps, and similar technologies. I got to work with platforms like OpenShift in an early stage. After experiencing that, I've been hooked on Cloud Native ever since.

Bart: In the ecosystem, things move very quickly. We do monthly content analysis reports, always asking people about what they find most interesting. What kind of content works best for covering Kubernetes topics? Is it diagrams, videos, blogs, or tutorials? What content would you like to see more of in terms of emerging technologies? What do you think is missing? Where are the gaps in the market?

Tanat: If you ask me about that, I feel like this is a new trend for the younger generation. Sometimes, quick infographics that can share or explain a hard concept in an easy-to-digest way are still lacking in the industry. I would love to see more of these in addition to blog posts and talks.

Bart: If you could go back in time and share one career tip with your younger self, what would it be?

Tanat: I tell myself to write more and share more knowledge because I think it helps you crystallize your ideas of what you want to share. It also exposes you to opportunities such as this one.

Bart: As part of our monthly content discovery, we found an article you wrote titled "The Karpenter Effect, Redefining Our Kubernetes Operations". The organization where you work, Adevinta, operates at massive scale with over 2,000 Kubernetes nodes across your ship platform. Can you give us some background on what led your team to explore AWS Karpenter as a replacement for your existing Cluster Autoscaler setup?

Tanat: So at our scale, managing over 2,500 nodes across 30 clusters with our previous setup, we used EKS Managed Node Group and Cluster Autoscaler. We were struggling with operational frictions. The biggest pain point was the tight coupling between the control plane and the nodes, which made version upgrades very brittle and noisy.

Beyond that, instance flexibility was a constant headache. Each node group was locked to a specific set of instance types, requiring significant maintenance effort to keep them up to date. When changing the node group configuration, you have to refresh all the managed instances, causing additional disruption.

Overall, the operation was quite brittle, inflexible, and at our scale, not sustainable. When I encountered Karpenter, I saw it showed good potential to solve our pain points.

Bart: So, Tanat, in our previous conversation, you discussed the complexities of Kubernetes upgrades and the move from blue-green to in-place upgrades. How do these upgrade challenges specifically drive the decision to adopt Karpenter, and what transformation did you see?

Tanat: As we discussed last time, transitioning from blue-green strategy to the in-place strategy saved us a significant amount of time because we don't need to provision the cluster, coordinate and switch traffic, and so on. However, the tight coupling between the control plane and the data plane was still a challenge.

We viewed the upgrade as a massive synchronous event. If anything failed during the process—such as an EC2 instance shortage, which happened more often than you might think—the entire process would roll back. Even if 100 nodes were already upgraded, a failure would trigger a complete rollback. The upgrade was incredibly stressful and forced us to do a capacity reservation before each upgrade, which added a lot of toil to the team.

Karpenter completely transformed this by decoupling the control plane and data plane upgrades. Now we just upgrade the control plane, which takes roughly 15 to 20 minutes. Once that's done, Karpenter automatically detects the new AMI version and handles the node rollout in the background asynchronously, with the capability to also control the pace of the rollout. The process becomes almost hands-free with more control.

Bart: Now, pod disruption budgets seem to play a crucial role in your Karpenter setup. How does Karpenter's handling of pod disruption budgets differ from what you experienced with cluster autoscaling?

Tanat: The difference is quite clear. With our previous setup, we had a hard deadline for the node running, even with the Pod Disruption Budget (PDB) set. If the pods couldn't be evicted within 15 minutes, the node would be terminated anyway, causing unintentional downtime for users. Karpenter, on the other hand, fully respects the PDBs. It will wait patiently until all the pods are evicted safely before terminating a node. This is great for reliability.

However, it has its own trade-off. If you configure the PDB too strictly or misconfigure it, it could block an upgrade indefinitely. This means we have to be more proactive in monitoring the rollout progress and identifying any stuck nodes.

Karpenter's node disruption budget feature allows us to control exactly how many nodes can be disrupted in a given time frame, providing full control over stability that wasn't possible in the previous setup. For example, we configured it to allow only one to three nodes to be disrupted every 15 minutes. If no PDBs can be passed within this window, the 15-minute slot will be skipped entirely, preventing node disruption.

This flexibility also allows us to implement niche requirements, such as allowing node replacement only on weekdays during working hours.

Bart: Instance selection flexibility is a major Karpenter feature. How did moving from explicit instance types to instance requirements solve the exhaustion problems that you were facing?

Tanat: This was one of the biggest wins from Karpenter because with EKS Managed Node Groups, we had to maintain an explicit list of instance types per node group. If a given type had a shortage during an upgrade, the whole process would fail. We were constantly burdened with keeping this list up to date. With many node groups, each being strict about size, we had to maintain a separate list for each.

Karpenter flips this around by allowing you to define not specific instance types, but requirements like CPU architecture, minimum and maximum CPU and memory sizes. Karpenter then picks the cheapest available instance type that fits the need to schedule pending workloads. This not only mitigates the risk of instance shortage but also means we automatically adopt newer, better, and cheaper instance types whenever they become available.

Additionally, we could define a multi-purpose node pool with such flexible sizing that it can fit workloads we normally find hard to predict. In the past, if a team requested a workload and we didn't have a configured node group, we had to work and provision a new node group. With Karpenter, it's much more flexible.

Bart: Bart asks: Tanat, you've created tailored node pools for diverse workloads like GPU jobs and monitoring stacks. Can you walk us through how Karpenter's CRD-based approach enables this flexibility?

Tanat: Defining a new node pool in Karpenter is as simple as creating a new resource in Kubernetes. We can easily define constraints like instance family, architecture, and more importantly, which taints and labels they should have to isolate them from the rest of the pool. For example, we have a dedicated node pool for GPU, ensuring that only our machine learning jobs can run to control the cost. We also do the same for Prometheus, which generally needs higher memory instances. We could specify only M family for this node pool. By having a dedicated node pool, it helps isolate workloads and reduce the noisy neighbor problem. In the past, each of these use cases needed to be provisioned in infrastructure as code and required more configuration for taint selector and instance type selection. In Karpenter, creating a new node pool is straightforward—it's just a YAML file.

Bart: And Kyverno integration seems to be a key part of your implementation. How are you using Kyverno policies to enhance the Karpenter experience for your users?

Tanat: Kyverno plays an important role in our setup as it acts as a smart automation and safety layer on top of Karpenter. We use it in two main ways. First, to simplify our user setup: instead of users needing to know which node selector or toleration to use for a node pool like GPU, we provide an interface where they can add a simple annotation to their workload. Our Kyverno mutating policy then sees the annotation and automatically injects the correct configuration for node selectors and tolerations to place these pods in the right node pool.

We also use Kyverno as a guardrail. Since Pod Disruption Budgets (PDB) are critical for ensuring stability, we have an admission policy that validates the PDB configuration. This provides a default configuration for users while preventing it from blocking the completion of a node drain.

Bart: Security patching through AMI updates is critical at scale. How has Karpenter automated this previously manual process?

Tanat: Karpenter can automatically detect when a new AMI version is available, including new security patches. It marks all nodes running the old version as drifted and begins to replace them gradually while respecting policies. This turns a manual, tedious process of tracking new AMIs into a more automated one.

In my team, this topic is somewhat controversial with two main strategies: using the latest AMI or version pinning. Using the latest AMI means you don't have to track versions, but we've encountered issues like a GPU AMI bug that prevented new nodes from coming up. With version pinning, you need to track versions, and some teams use external automation to manage updates. The benefit is the ability to test new versions before production deployment.

Regardless of whether you prefer pinning or not, Karpenter's ability to roll out nodes in a more controllable way makes applying security patches less painful and allows for more frequent updates.

Bart: You achieved significant cost savings of 30,000 euros per month by migrating from Intel to AMD instances. How did Karpenter facilitate this transition without manual intervention?

Tanat: We had been planning to switch from Intel to AMD instances even before the Karpenter project. We did a cost-saving analysis, and the numbers were significant—around 30k per month. However, the project was always postponed because manually replacing thousands of instances was a huge effort.

When we migrated to Karpenter, we simply defined our instance requirements for the node pool. By default, since Karpenter picks the cheapest instance that meets the criteria, AMD instances were naturally selected over Intel. When we drained the old Intel nodes, Karpenter automatically started provisioning AMD instances. By the end of the process, we had essentially accomplished the project and were running most of our instances on AMD.

Bart: Scheduling latency became a challenge with Karpenter's efficient node packing. Can you explain the over-provisioning strategy you implemented with low-priority placeholder pods?

Tanat: The driver for this is that Karpenter itself is excellent at bin packing pods to maximize resource utilization. It will proactively deprovision empty nodes to save costs—a process Karpenter calls consolidation. However, there is a downside: it leaves very little room for scaling pods if there is a sudden burst in traffic.

After migration, our users reported that when their services were hit with a traffic spike, the tail latency of their services increased as new pods were scaled and waiting for nodes to be ready. Our solution was to deploy a set of low-priority placeholder pods. The intention is to have these pods reserve capacity on the cluster. When a higher-priority user workload needs to be scheduled, Kubernetes immediately evicts our placeholder pods to free up space for these more important workloads, which speeds up the scheduling time for our users.

Bart: Now, the cheapest first selection led to some unexpected performance issues with M5A versus M6A instances. How did you identify and then resolve this problem?

Tanat: After the migration, some users reported performance degradation of their services. We did some digging and discovered that Karpenter was choosing the older M5A instances over the newer M6A instances because they were slightly cheaper. The paradox is that the newer generation often have better price performance, around 15 to 20 percent. Even if they cost a bit more, they supposedly get more work done, meaning they need fewer pods to accomplish the same load. But Karpenter picks the cheaper option.

To fix this, we created a layered generation-based node pool with the help of the weight concept. We now have a primary node pool with a higher weight that is restricted to the latest generation, which forces Karpenter to always prefer the newer, more expensive, and more performant instances. We also keep a lower priority pool with the same configuration but strictly limited to the older generation as a safety net in case of capacity shortage for the newer generation, which typically happens when a new generation has just been released and global capacity is still limited.

Bart: Karpenter provides extensive metrics that weren't available before. What new insights and observability capabilities have been most valuable for your team?

Tanat: The observability we have gained from Karpenter has been huge for our operations. The most valuable metrics are node drift status, which is critical for tracking upgrade and rollout progress. We also track metrics around consolidation actions, which help us understand how Karpenter is currently optimizing the cluster. We closely watch the instance type distribution and the number of instances per node pool. Overall, this data allows us to make data-driven decisions about capacity, cost, and stability—information that was not easily obtainable in our previous setup.

Bart: Looking at your entire migration journey, what would you say are the most critical lessons learned for other organizations considering Karpenter at scale?

Tanat: First, start with data. Make sure you have visibility over how things are moving inside the fleet, which is key to making further adjustments. We noticed several issues in the first phase where the disruption pattern from the dashboard helped us tune the configuration until it was right.

Next, get the Pod Disruption Budgets (PDBs) right because they are there for stability, but they could also block you if not configured properly. Plan your scaling patterns because you might need an over-provisioning strategy like we did if you have bursted workloads.

Understand that the cheapest option isn't always the best. Be prepared to guide Karpenter in node selection to balance between cost and performance. Finally, don't forget about leveraging policy automation like Kyverno to simplify your user experience and implement guardrails around usage.

Bart: Now, Tanat, what's next for you?

Tanat: I'm still writing blog posts about SRE DevOps incidents. Lately, I've been spending some time on engineering metrics related to LLM usage. Stay tuned for some content around that.

Bart: If people want to get in touch with you, what's the best way to do that?

Tanat: I'm very active on LinkedIn. That's probably the best place.

Bart: Perfect. Thanks so much for your time today. It was wonderful having you back on the podcast. I've got a good feeling we'll probably be seeing you here again. Take care and have a great day. Cheers.

Tanat: Cheers. Thank you, Bart.