Bart Farrell: Today on KubeFM, we're talking about one of those platform engineering decisions that starts with a very specific problem. In this case, workload identity in Amazon EKS. What quickly becomes a much broader discussion about abstraction layers, operational ownership, and how platform teams evaluate new Kubernetes projects. Fabián, who's been a guest on KubeFM before, recently documented his team's journey from KIAM to EKS pod identities, comparing Crossplane, custom controllers, ACK, and KRO along the way. Rather than asking which tool is best, The conversation is really about how experienced platform teams make architectural decisions that balance flexibility, operational complexity, and long-term maintenance. Kubernetes requests and limits are easy to guess and hard to get right. Set them too high and you risk capacity. Set them too low and you risk throttling, OOM kills, or downtime. LearnKube has put together a free set of resources to help. A book, interactive calculators, and a cheat sheet for getting CPU and memory values right. You'll learn how to identify over-allocated containers, and decide what is worth changing safely. Download them at learnkube.com slash kubernetes-rightsizing. Now, let's get into the episode with Fabián. You're tuned in to KubeFM. All right, Fabián, welcome back to KubeFM. What are the three emerging Kubernetes tools or features that you are keeping an eye on?
Fabián Sellés Rosa: I'm not following any specific tool at the moment, but I'm closely following gang scheduling implementation, which is composed of three features, mainly, which are PodGroup API, Workload API, and I think it's called Dynamic Resource Allocation, if I don't remember it wrongly. Those three composed together to the Kubernetes scheduling, which I think is pretty interesting for the new Kubernetes scheduling and for the AI workloads.
Bart Farrell: Now, you've done an episode with us before, but we still want to know for folks that haven't listened to that previous episode, which they should check out, can you tell us a little bit more about what you do and where you work?
Fabián Sellés Rosa: I work in Adevinta. It is a company that essentially sells, it's a retail company across Europe. It has a lot of brands across Europe. Currently, I'm the platform engineer at Adevinta. I'm the tech lead of the runtime team. My team is responsible for managing Kubernetes at a scale that serves as a platform, as a service for Adevinta. So we host the workloads of the Adevinta developers from queue consumers to Pods, HTTP APIs, jobs, machine learning trainings, machine learning servings. And right now we have had up to 30 clusters and a lot of nodes across four regions.
Bart Farrell: Okay. And how did you get into cloud native? What was the beginning of your journey like?
Fabián Sellés Rosa: I started not in the cloud native world. I started as a pure sysadmin, then I was exposed to this Heroku, I think it was from Heroku, the 12-factor app manifesto, where I learned about the origins of the cloud-native initiative. Then I started in Adevinta to deploy a Mesos cluster and work with Mesos. Later on, I derived to start working with Kubernetes, where I wrote the platform and the went from zero clusters to 30, where I learned what Kubernetes means on the whole ecosystem, the CNCF, et cetera.
Bart Farrell: And the Kubernetes cloud ecosystem moves very quickly. How do you stay up to date? What works best for you?
Fabián Sellés Rosa: Well, it's hard to stay up to date, to be honest. But what I try to do is, well, I read, obviously, the Kubernetes blog. I think it's the one interesting source of keeping me updated. I skim over Kubernetes Enhancement Proposals in the GitHub repo. I think it's interesting to at least skim over those. I read a Medium posts. I also watch KubeFM and KubeArchitect.
Bart Farrell: Much appreciated. And if you could go back in time and share one career tip with your younger self, what would it be?
Fabián Sellés Rosa: Well, it will be to be consistent on learning and self-improve every day and later share what you learn. I think that's a tricky one because it's not easy to always have time to keep investing on your own learning and your self-improvement and also sharing what you have learned. But that will be definitely my tip.
Bart Farrell: Fantastic. Now, for today's conversation, we want to talk more about an article that you wrote titled, From KIAM to EKS Pod Identities, Evaluating Crossplane, Custom Operators, and KRO, or K-R-O for folks that are familiar with it. So, SCHIP is the platform context behind this story, but listeners may not know what it means in practice. Although we have had multiple guests from out of Adevinta join us on the podcast. We've talked about SCHIP before. But for people that are new to it, Fabián, can you walk us through what is SCHIP and what kind of teams and workloads it supports? And what does the operating environment look like at that scale?
Fabián Sellés Rosa: Well, SCHIP is essentially a Kubernetes distribution. It's our platform as a service inside of Adevinta, a web that we have built to serve multiple teams and multiple brands across Europe. Mainly, we host HTTP APIs, queue consumers, machine learning training and serving models, or jobs. We host those across four regions and multiple, many multitenant clusters. The fleet varies in a scale over time. So we have had from zero cluster to 30, from 5K requests per second to 300K. So it varies, but essentially this is what it shows.
Bart Farrell: And now in a shared Kubernetes platform, applications often need AWS permissions, but handing broad permissions to the underlying nodes creates a security boundary problem. How did your platform originally give workloads access to AWS credentials?
Fabián Sellés Rosa: Good question. So Well, for starters, we need to think about that the SCHIP was initially created and introduced in 2016, so 10 years ago. At that moment in time, we didn't have native solutions for identity for containers. Amazon didn't have Pod identities or IRSA and we needed a way to inject credentials into containers. So the credentials to, in order for applications to interact with the Amazon services like S3 or the DynamoDB or RDS, right? As many know, in Amazon Web Services and other cloud, you can assign an IAM role, which is a specific set of credentials, a set of permissions, to an instance, a node, that's called an instance profile. But we cannot do that when we are working with containers because essentially, in a node, you can have hundreds of multiple containers sharing the same set of credentials, which is a security issue. And we had as a goal that SCHIP needs to be multitenant and to leverage economies of scale. We also wanted to avoid situations like, okay, we have in this end node an application from marketplace A that has access to Database B, and then application from marketplace B can access that same database. So we need a solution to prevent that sharing of credentials. At the moment in time, there were a couple of projects that were pretty famous and known. kube2iam was the most known, I think, and KIAM, which is the other one. But essentially, they were Pretty similar. The idea is that they have a deploy app proxy between the container and the instance metadata service from Amazon. So they use the same mechanism that nodes uses to inject credentials to expose that to containers. So a container when asked, okay, give me credentials, instead of giving the node credentials, it will give another set of credentials that are restricted to the container, right? Which is KIAM because we thought it was better. well, a little bit more secure than kube2iam and well architected because it has differentiated agents and servers. And that's better because essentially kube2iam all of them share the assumed role part, which means that all your nodes need to have access to role credentials. In KIAM, you can delegate a specific set of nodes for that, which is better because essentially only a small amount of nodes. will have access to request roles and the rest just needs to use them. And also we need to understand that at that point in time, Kubernetes was not a thing. People still were not up to containers. So people usually have an Amazon account where they have hundreds of instances, hundreds of microservices running. So containers was a little part of the equation. And we need something that connects containers to the Amazon environment where the, let's say, production load we're running. So in that scene, what we felt is okay, let's use It KIAM will inject our credentials into the container. Then what we will do is that we will offer an interface for customers. Our interface will be okay. We will define an IAM role. That IAM role people will need to trust in their tenant account, in their Amazon account, in their site. And then KIAM will inject the right credentials in the container securely because essentially that set of credentials will be only used for the container. And customers, users, can define which role they want to assume using a specific annotation. And also, we have other restrictions or security mechanisms that ensure that marketplace A cannot access marketplace B. Like, at namespace level, we can ensure that only a certain pattern of roles can be assumed.
Bart Farrell: Now, that approach can work well for a long time. especially when application teams are used to the interface. But platform dependencies sometimes fail when the surrounding assumptions change. What happened that made this identity layer urgent to replace?
Fabián Sellés Rosa: So it all started at Friday evening, like the good stories. So it was Friday evening. We got a page. We thought, OK, what is going on? We checked the page, and then it was coming from one of our applications that actually was the one generating more revenue. And we looked into the page, and we thought, OK, it looks like it's not getting credentials. IAM credentials. We were like surprised because this typically don't change and don't fail that often. So, okay, we were thinking about what was going on. We were doing a routine control plane upgrade. We do that from time to time. It was tested with interesting dev, everything was fine. And then we thought, okay, if it's not the control plane upgrade and nothing has changed, what the heck is going on? Right. We know at the moment in time, it was the beginning of this year, KIAM has been unmaintained for a long time. We have always been thinking about replacing it, but life happens, and you have all the other priorities to perform, and you don't always have time to invest in all the other things. Looking into the page, we saw unauthorized from KIAM, and we thought, okay, why now is KIAM unauthorized? We just think, okay, maybe it's the control pane upgrade, so let's refresh the Pods, let's see what happens. We refresh the Pods. and everything started working again. And we thought, okay, maybe it's transient. So maybe it's something related to the control plane. Maybe it has cache and AP or something. It's working. It's fine. Let's mitigate the incident. Let's focus on that tomorrow. Three hours later, we got paged again. And when we got paged again.
Bart Farrell: Once the immediate incident was contained, the next step was designing a proper replacement rather than just swapping one component for another. What did the new solution need to preserve for application teams? and platform owners?
Fabián Sellés Rosa: Well, we need to ensure that the interface is respected as much as we can. So essentially, we need to ensure that the application owners can still specify which roles they want to assume using annotation or using an object or something. application owners need to ensure that they can define which roles can be assumed not in the namespace or it's in potentially, if possible, the same condition as before in KIAM. And preferably, the mechanism, the way we inject credentials should be supported, right? Should be maintained. If it's by Amazon, it's even better.
Bart Farrell: Now, moving to a supported AWS mechanism sounds straightforward from the outside, but identity is part of the application contract. What changed for application teams when you moved away from the old model?
Fabián Sellés Rosa: Well, Prior to this incident, looking into the details about what went wrong, we know there were Amazon native alternatives like IRSA and EKS Pod Identities, but they changed a little bit the interface content. So the first one is that in KIAM, use a annotation to select which role people want to inject in their pods. Using Pod Identities, you need to use a service account, an Amazon API call that binds service account to a specific target role. And then also tenants in their side, they need to trust Amazon Pod Identities role or something else, sorry. So obviously, this means that the additional work for application owners and they change a little bit the interface. And while we wanted to go to the model, we need to do it in a way that is seamless to application owners.
Bart Farrell: And before choosing an implementation, You defined evaluation criteria rather than jumping straight into tooling. What did you care about when comparing possible approaches?
Fabián Sellés Rosa: Well, I wanted to make sure that I picked the right choice. And in order to pick the right choice, I need to think about what are the characteristics I wanted to keep. So first of all, I wanted to be flexible because, OK, now I can maybe fix my current identity solution, but maybe I can reuse it for something else later. It's also important that it's mature, if it's established or running in production, or it's just developing. And if it's easy to operate, because as a platform team, you always have struggling priorities, you always need other things to do, so you need to reduce operational workload as much as you can. And if you can reduce the ex-operations, you can increase the ex-operations even better.
Bart Farrell: One possible direction is to build on an existing platform abstraction system rather than writing the reconciliation logic yourself. What did that option look like in your evaluation?
Fabián Sellés Rosa: So KIAM essentially is an operator, a controller. What we need to do is into something that replace KIAM, writing a custom controller. There are many options to implement custom controllers. One of them is probably well known is Crossplane. It is a widely used and graduated CNCF project that essentially allows you as a platform team to create your own abstractions, your own CRDs, your own controllers. In broad words, what it does is four things. It helps you to define your own CRDs. It helps you to manage non-native and native. with Kubernetes resources, like for the Identities, for instance, or by the IAMRole in this case. And allows you also to react to events and package things together. So it looks like a compelling alternative.
Bart Farrell: When platform teams introduce a new abstraction, the user-facing API matters as much as the back-end implementation. How did you think about the interface that application owners would actually use?
Fabián Sellés Rosa: Yes, well, in this case, for instance, I When trying Crossplane, what I decided is to use a native standard CRD. I did that because I wanted to compare apples to apples, to compare Crossplane implementation to other potential implementations, and also because I wanted to hide complexity for users. I don't want users to know if I have used Crossplane or not, or a specific field or not. I only want them to focus on the immediate problem, which in this case is okay. I want to expose an application. I want to inject credentials for an application, and I need to define which service account or application I need to inject the credentials, and which IAM role I want to assume to have the credentials in the environment.
Bart Farrell: Existing ABSTRACTION FRAMEWORKS can save a lot of custom code, but they also bring their own control plane and debugging model. Where did the trade-off show up with that approach?
Fabián Sellés Rosa: Well, Crossplane is super flexible and powerful, but it has many components that can make debugging little bit challenging. So when you have issues, you have something breaking, you need to inspect the Crossplane object itself, like XRD, compositions, you need to check the logs of the providers and the Crossplane logs. So it's very hard to compile everything and make your mind about what is happening. It's possible, it's not impossible, but it's very difficult. But on the On the other hand, Crossplane is very powerful and versatile. In this case, probably it could be possible to implement the use case, but also it's interesting that it will serve in other use cases or potential use cases in the future. If it was the choice, obviously I would need to invest. I would need to invest as a platform team in the observability and tooling to help with the guidance.
Bart Farrell: Now, another possible direction is to own the reconciliation logic directly. If you take that route, what does the architecture look like?
Fabián Sellés Rosa: So essentially, you roll up your sleeves, start coding, or in these days, you just chat with Claude, depending, and write your own custom controller. There are many frameworks out there that you can use. For instance, you can use a Kubebuilder or a Operator SDK to write your own controller. There are low-level libraries like controller-runtime, which is used by the Kubebuilder itself. So essentially, you need to write your own reconciliation loop. to interact with the APIs, in this case, interact with the pods when it arises, with Amazon Pod Identities to create a new Pod identity, ServiceAccount etc. And you need to define your own CRD as a spec, and implement the reconciliation based on it.
Bart Farrell: Now, owning the code gives you control, but it also changes the maintenance contract. What would your team have been responsible for if you had chosen that path?
Fabián Sellés Rosa: Essentially, we would need to be responsible for everything. since keeping it running, keep it stable, patch, which means any API change you need to adapt, any security dependency update you need to bump it, you need to take care of anything that changes in that space.
Bart Farrell: After looking at those extremes, you searched for an approach that reduced custom code without bringing too much operational weight. What did you find?
Fabián Sellés Rosa: Yes, so at this point, I like very much the flexibility of Crossplane. but I also like very much the simplicity of maintenance of writing my own custom controller. So I wanted something in the middle. So that's why I think KRO, Kubernetes Resource Orchestrator, fits the gap. Kubernetes Resource Orchestrator is a project that allows you to define CRDs using a specific RGD from KRO, which is a ResourceGraphDefinition It's like a template for CRDs, where you define an object. that allows you to explain, okay, I want to create CRDs that look like this. And then KRO will implement the CRD and templatize the result. And also reacts to the CRD you create, so it's kind of a reconciling itself. And you can feel like Helm, but using Helm for CRDs, but using Common Expression language.
Bart Farrell: Defining a higher-level Kubernetes resource is only half the problem. Something still has to represent and reconcile the AWS side object. So how did you bridge that gap?
Fabián Sellés Rosa: Yes, as I was mentioning, KRO is pretty specific to define CRDs in CEL, but it doesn't implement the logic of binding the creating or calling the pod Identity API call, not doing the actual work. So I could have used, for instance, Crossplane to implement that using also KRO as an orchestrator, but that looks like a little bit complex as well. So I was looking for another entity. That's where I found AWS Controllers for Kubernetes, which is a set of controllers maintained by AWS that exposes AWS APIs as Kubernetes objects and reconciles them.
Bart Farrell: Operational ownership was a major part of the decision. How did managed controller capabilities? Sorry. How did Managed Controller Capabilities in EKS change the trade-off for this design?
Fabián Sellés Rosa: So, well, for instance, recently EKS introduced capabilities, which is essentially managed cluster applications for Amazon. So it's similar to existing EKS add-ons. And the initial capabilities offered were Argo CD, KRO, and ACK. So I have the two options as managed. which makes KRO plus ACK, a good alternative, because essentially I don't need to manage those components in my platform. The AWS will do it for me, for a fraction of the price. This reduces my operational overhead and make the platform team focus on the important matter, which is solving the problem.
Bart Farrell: And now, like with every case, no option was perfect. Once you had the low toil design working, what limitations or risks did you still have to accept?
Fabián Sellés Rosa: Of course, so KRO is heavily under development, so it's still a beta project. So some key features are missing. One significant, for instance, issue I think is very known is that when you generate a CRD from an RGD, it works, it's pretty well defined, but then when you change it because you need to add something, it doesn't allow you to version it, which means all the objects that KRO manages will be changed at once, which could have, depending on the case, large impacts. I think it's not possible, at least that I know, to bubble up the status from managed objects to CRDs managed by KRO, which means if I bind my pod Identity object using ACK and it's bound, how can I get that status bound towards the main CRD like IAMRole that it creates? I cannot do that. And also, since this is managed by AWS, there are also obviously observability gaps. It's not super very easy right now to find out what's going on inside Amazon. managing EKS capabilities. So when KRO fails or ACK fails, you have logs and not much more.
Bart Farrell: One important part of the old model was authorization. Namespaces could restrict which IAM roles workloads were allowed to request. How did you preserve that security boundary in the new design?
Fabián Sellés Rosa: Yes, you see in KRO, plus, ACK, we have met two of the principles we wanted, two of the features we wanted, which is we kept the interface of application owners to define. which roles they want to request, and we have it maintained by AWS. But then we still missed the third one, which is, okay, we need to ensure that IAM credentials can only be assumed for the ones that the application owners define. So we didn't have anything inside KRO, plus ACK that allows us to implement that. Then I look into what I currently use in SCHIP in our platform, and then we found out that we use Kyverno. as a policy controller. And this is actually a policy. We don't allow Team A to write a CRD IAMRole that includes this specific regex pattern because it's outside your namespace. So conceptually, this is the Kyverno policy that we implemented. This is kind of the straightforward. So Kyverno gets the annotation from namespace that is a regex, says, okay, I allow, in this namespace, I allow IAM roles to be assumed when you try to create an IAM roles here with an IAM role that doesn't match the regex, it will forbid it. So it prevents other people trying to assume roles in a namespace Because in SCHIP, namespaces are team-mapped So a namespace belongs to a team and it's always going to be.
Bart Farrell: Now, Fabián, looking back, the outcome was very specific to your platform constraints. How should another platform team think about choosing between these kinds of approaches?
Fabián Sellés Rosa: I think it's important that first you define what's your target solution and the evaluation criteria because it's pretty easy to fall under the shiny object syndrome. So I'll look into the new shiny tool. And I think spending time on defining the evaluation criteria and the target solution is important because it allows you to evaluate different alternatives according to the criteria and making an informed decision.
Bart Farrell: Now, what's next for you?
Fabián Sellés Rosa: So that's a good question. I don't know. Actually, probably this is one of the last posts writing for SCHIP because SCHIP has been decommissioned. So probably looking for another project to learn a lot and try to share with people. So I don't know. The future is bright, I think.
Bart Farrell: Okay. And how can people get in touch with you?
Fabián Sellés Rosa: You can find me on LinkedIn, Twitter. I don't know. Search my name. You will find that way.
Bart Farrell: And like I said, we've already had you on the podcast before. So you're well established in the KubeFM ecosystem. That being said, thanks so much for joining us for a second episode. And I'm sure there'll be an opportunity in the future to do a third. Keep up the amazing work and we'll be in touch. Take care.
Fabián Sellés Rosa: Thank you. Bye-bye.
