How Structured Authentication Config changes Kubernetes auth

Bart: When it comes to Kubernetes, one of the ways to have influence is to be involved in a special interest group or SIG. There is a SIG called SIG-Auth related to authentication. And in terms of talking about authentication configuration, things like JWT tokens, it's important to be in touch with someone like Maksim. He's in this episode of KubeFM telling us all the things that you need to know when it comes to authentication configuration, all the things that go into that, as well as updates that came in the last version of Kubernetes 1.29. This episode is brought to you by Learnk8s. Learnk8s is a training organization that offers courses to help folks level up their Kubernetes career, learning all the things that they need to know. 60% of the courses are hands-on and practical, and 40% are theoretical. You can join them in person or from the comfort of your own home. You'll have access to the course material for the rest of your lives, and they are given by amazing instructors that can help you improve your Kubernetes skills. Let's get to the episode with Maksim. All right, Maksim, welcome to KubeFM. To get started, if you were to install three new tools on a brand new Kubernetes cluster, which ones would they be and why?

Maksim: Okay, this question is a little bit difficult because there are, so if you're familiar with CNCF landscape, there are myriads, hundreds of different tools that helps you do X in your Kubernetes clusters. So there are a lot of them, but probably the first thing I would like to install will be some tool to manage my cluster configuration, probably some tool that I can use to manage my cluster configuration. GitOps thing like Argo CD or Flux CD because I'm not a big fan of GitOps but at least this will provide you necessary capabilities to manage your configuration. So let's go with the first thing, some GitOps tool. Yeah, so this will be the first thing. So the next one, probably the CNI plugin, because, you know, CNI makes networking in your cluster a lot easier. Yeah, so this is a choice by default. So I will... Usually we tend to choose between, I don't know, Cilium or Calico or other CNI plugins, but my pick will be Flannel. This is maybe not an obvious choice, but this is a good piece of software that proved to be doing its job. So I would like to mention the Flannel just because it's the old player. So this is the second tool. And the third one probably will be, I don't know, Dex, because I'm the maintainer of Dex. So yeah, let's make it Dex, the third choice.

Bart: Okay, good. I'm sure we'll talk about that a little bit more later on. That being said, tell me more about what you're doing, where you work, who you work for.

Maksim: So currently I am working for the company called Palark. We are based in Germany and we are a service company. So we provide the thing called DevOps as a service. So we provide you a fully functional DevOps team with the project manager, with team leader. And then they can do. whatever you want in terms of DevOps, in terms of automatization, in terms of supporting your infrastructure. Usually I say that we do digital transformation. So if you want the digital transformation, but you don't know how to do this, we can help you with this. And you if you don't have enough hands to do the job, we can do it for you. And we also can. help you with your production 24-7. So we also have support. That's it. That's what we are doing. And I'm personally in charge of developing our internal tooling. So our tools for like a common tools to provide for our clients. And I'm in charge of building our internal Kubernetes-based platform. That's it. So usually I... search for the best open source solutions on the market and plan how to integrate them into our platform for our clients this is uh like mostly this is my job okay good yeah and so this is this is a about work but in my free time i was uh because i for while i'm working i'm more managing people than writing the code so in my free time i like to write the code Yeah, so I like to work with the community, with open source projects, so I contribute a lot to Cloud Native Computing Foundation. I also like to contribute to Kubernetes, mostly to things related to authentication. I don't know why, it's just because of some historical reasons, I don't know. Yeah, I also contribute some features to Kubernetes authentication in combination with team members of special interest group Auth, so Auth announced that. all this stuff and i am i also i am a part of the dex team i am one of the core maintainers dex and open id connect providers plugable connectors we will make i hope we will be able to talk about this later so that's what i'm doing now

Bart: Fantastic. And how did you get into cloud native? You know, like you said, you talk about digital transformation, DevOps as a service, maybe platform engineering as a service in the future. But how did that journey start for you?

Maksim: The first thing, like the first time I saw cloud native technologies was when I started working in my current company. So before that, I didn't have a chance to work with CloudNative. I usually worked with Bare Metal servers. So yeah, this was the first time. And I was amazed because my current company is a service company. We have many clients with many environments in different clouds, in Europe, in other parts of the world. So many, many things to learn, many things to... research to understand. And I was so amazed that... So at first I was working as a DevOps engineer, so it was my first job, my first position in this company. But after five months, my team lead, he noticed that I am good at researching of new technologies because I was so amazed by the number of software, by the number of... clouds and all the things. So I liked to research everything about them to understand these technologies. And after that, the management of the company decided to move me from the DevOps team to R&D team. Yeah, so mostly I used to work as an R&D engineer.

Bart: Got it. And with that in mind, how did you learn Kubernetes? You know, in terms of resources, books, blogs, podcasts, like what's been your strategy in terms of learning Kubernetes over time?

Maksim: I have many books in my house on my shelf, but there is no book about Kubernetes. I've never had a chance to read one. But Probably I can say that I learned Kubernetes by practicing, by working, by researching incidents with Kubernetes, by understanding why is this thing working that way. Because I think the practice is what can help you understand more than you can read in books. For example, yesterday I found out that... Taints that you can put on your node in the cluster, they are unique by the key and effect. And it was so bizarre to me, like, why? Because tolerations, tolerations on your pod, tolerations match tains by the key, effect, value. But taints are unique only by key and effect. Why is this? I was searching history in GitHub. I was finding blame, searching for a pull request, but I didn't have a chance to find the root cause of this. Yeah, but this is just a thing. And I don't think you can learn it by reading the book or documentation only by working with the real production.

Bart: Sounds good. And makes sense that hands-on experience is difficult to match through other means. With that in mind, also, you know, Kubernetes or not, if you could go back, you know, in time and give yourself some career advice about anything to perhaps, you know, help you to be a better manager or a better engineer, what advice would you give to your previous self to help you in the past?

Maksim: This is a tough question because I am... pretty much satisfied with my career so far and i honestly i don't know maybe so the the one the one only one tip so maybe the one will be about like not hesitating to ask for help because it can save you like a lot of time, a lot of, I don't know.

Bart: Pain and suffering.

Maksim: Yeah. Yeah. So it's better to ask community for the solution because it can save you a lot of, not only time but like the decrease your stress level.

Bart: Absolutely. I completely agree. Make community part of the solution. Don't suffer in silence. And you'll find yourself later on helping other people by answering their questions and, you know, paying it forward that way. That's great. For today's episode, we want to focus specifically on, you know, structured authentication config in Kubernetes version 1.29. So you mentioned you're part of SIG-Auth. Can you just tell me again, you know, what do you focus on there?

Maksim: Initially, the idea of this feature... came from one of the team leaders of the SIG authentication. And they were searching for the person who can help with the implementation or with writing the KEP. And firstly I thought, why wouldn't I take the part and try something new for myself? So I decided to participate and... This feature is the most significant feature, I would say, that happened with authentication in Kubernetes for past, I don't know, five years or something like this. Yeah, so basically the the idea of structured authentication config is that there are a lot of flags that you can use to configure your API server. So you can go and check yourself by typing minus minus help in your terminal and you will see like I don't know, hundreds of flags you can set for your Kubernetes API server. And there are also flags to configure authentication. For example, you can configure authentication through OpenID Connect server by setting some flags there. And this is not flexible enough. So because you cannot set... many different options using the flex because it's cumbersome approach, I don't know, it's not that that simple to set some options. Yeah, and this was the first reason to migrate from flex to some structured authentication config, this was the first reason. The second reason is that to change a flag, you need to restart the process of the API server. So it's not convenient because if you want to adjust authentication settings, you need to... So for example, if you have only a single master node in your cluster, this will be a downtime for you. Yeah, you don't want to have downtimes a lot. in your cluster. Yeah, it was the second reason. And the third reason to introduce this was to add all the features that were asked for the past five years, add more flexibility to configuration of authentication. So that's it. That's three points I would like to mention. And we went to implement this thing from this point.

Bart: And, you know, just for clarification, can you define what is structured authentication config and why it's so important?

Maksim: So, basically, structured authentication config is a configuration file with a defined structure that you can use to configure authentication. So it aims to replace all the flags you previously used to configure it with a single flag that points to a config file with the defined structure. This is it. And in this structure there are many more capabilities than it was before. So this is what it is about. The main reason is to provide... the more suitable way for people to represent a configuration for their authentication. Something like this.

Bart: Good. Could you touch on the JWT compliant tokens, changing configuration dynamically, and CEL to match tokens?

Maksim: Usually we talk about OpenID Connect. uh as a as about like as a standard but uh kubernetes uh to authenticate users it's on it only checks the your GWT with some checks. At first it validates the signature of your token by going to OpenID Connect discovery endpoint, by finding the keys endpoint, by downloading the keys, and then by checking the GWT signature with these keys. So this is how Kubernetes authenticates tokens, like external tokens. yeah then after that it checks your client, for example, that the token issued for the particular client, the token is not expired, the token has the subclaim and all this stuff, the issuer claim, that issuer claim matches the URL you used to request the OpenID Connect discovery endpoint, all these checks. And only after this Kubernetes authenticates you and allows you to do a request. yeah this is what about openid but the the idea of GWT is that uh it it doesn't need to be specifically openly issued by the openid connect provider it can be any GWT and we can perform some additional checks so we can turn off uh the basic openid connect checks and write our own checks to authenticate the token so it doesn't need to be uh OpenID connect complaint anymore. This was the main idea.

Bart: You mentioned that the new change includes JWT-compliant tokens. How does this affect users or operators installed in the cluster? And were the previous tokens not compliant?

Maksim: No, no, no. You're right. Previous tokens were also JWT-complained, but because we want to support more providers and more formats of tokens, we don't call them OIDC tokens anymore. So we call them just JWT tokens. And I believe that we use OpenID Connect. only to authenticate users in the cluster, not for services and operators and other pieces of software. So to authenticate software in your cluster, we usually use service account tokens and service accounts, not OIDC tokens. So OIDC, usually for the external users, for cluster administrators, cluster users, editors to authenticate someone from outside of the cluster.

Bart: This change introduces the ability to have multiple authentication providers working together. Can you provide an example of what you mean there exactly?

Maksim: Yeah, so previously with the set of flags, you could only set a single OpenID Connect provider. But now with structured authentication config, JWT provider is now an array. And you can set as many as you want. You can set, for example, your staging OIDC provider and your production OIDC provider at the same time for the same cluster and authenticate using one of these.

Bart: But I guess the question is, how does this work? If we're thinking about, we know, we understand what is being replaced, but how does this affect products like Keycloak?

Maksim: I don't think it affects any products, but for example, previously, because you could set only single provider, there was OpenID connect servers like DEX, who used to act as a proxy OIDC provider. So in DEX, for example, previously, back in the days before structured authentication config, you could only connect a single OIDC provider to cluster. So you could connect DEX to your cluster. And in DEX, then you could define any number of other OpenID Connect providers you wanted. And this was one of the use cases for DEX, just to be a splitter for Kubernetes, to connect more than a single provider to Kubernetes. But now Kubernetes can do this without Dex, but I don't think it really affects. software like KeyClock. I think like just now you can connect two KeyClocks at the same time to a cluster. Then this is cool. A cool feature.

Bart: Good. Now the third change introduced in this KEP is the ability to check tokens with CEL. Can you give us an example of what this might look like?

Maksim: Yeah, this was the main idea behind the customization of token validations. Yeah, because CEL was introduced for a couple of, like, for half of a year, it was introduced to validate the custom resource definitions, and there were many ideas where CEL can be applicable. yeah so in one of ideas was to authenticate users and how so the first one was to i wouldn't say validate the token but i would like to say apply some authentication policies to the token like how how to authenticate the token for example we can add a custom check to check some groups in the token or, I don't know, to add the custom checks for expiration, for example. Yeah, so this is possible with CEL because CEL is flexible enough to do your complex checks because this is an expression language. Yes. And yeah, so, and then we decided to go further. not only use CEL to validate the claims of the token, but also to validate the final list of user attributes. So, you know, you can have many authentication policies to validate the incoming params, but you also would like to have some final validation that validates the user attributes that you got finally. Yes, so because user attributes, they are extracted from the claims of the token. Yes. And this is why so you can have some policies before. extracting user attributes using CEL. And you can also have some policies after authentication to validate the final result of the user, final user attributes. So this was our idea to validate the token additionally. And this is not the only place where we use CEL. CEL is also used to extract the claims of the token, like how to convert attributes from the token to user attributes of Kubernetes. For example, how to extract the name of the user from the token, how to extract groups of the token. And you can not only extract the groups from the token, but you can also do some modification, like to prefix all the group or to add suffix to filter the groups. like many different things with CEL. This is the second place. We call this thing claims mapping, how to map claims to user attributes.

Bart: Now assuming the KEP is merged in 1.29 as alpha, what's the roadmap for beta and GA?

Maksim: To be honest, I am not the part of the... implementation team, so I'm not working on the KEP implementation. And I would say that this is totally fine in Kubernetes. You can, so you don't need to be the person who implements the KEP if you took part of writing the KEP. And this is cool because in Kubernetes community, you can find your own place where you can. where you can contribute to the product and how you can contribute. Yes, so, but the roadmap is to implement all the features mentioned in the KEP and also to fix, to cover some additional use cases because, for example, to support multiple client IDs. Because now in alpha, as far as I'm concerned, multiple client IDs, they are not supported in the token. Something like this. So as far as I remember, we need to gather more feedback from users. So we want users to start using the feature and give us something. Yeah, and we also want to continue implementing the KEP. This is the goal for the beta release.

Bart: Good. And if people want, you know, you mentioned, you know, community and different ways of getting involved. If people want to, you know, give feedback about the KEP or possibly also get involved in the project to contribute, how should they do that?

Maksim: How can you be involved in the process? Yes. Yeah. So I started by. just writing some messages on Slack. There are Slack channels for every team in Kubernetes, so you can just join at least and see what people discuss, what bothers people. Some context of the channel. Then you can start asking questions or answering questions. As I see it, this is a good start. at least by sharing your knowledge. Yeah, and there are usually also regular meetings that you can attend. Basically, you can go to... So every two weeks, you can go to a meeting and listen to people proposing their ideas for new features or discussing some interesting bugs. And you can also discuss and take part in these meetings. And later, you can propose your own KEPs. like I did this year. So I also introduced another KEP for the another API change connected to authentication this way. So just by reading the chat, thinking about stuff and just one day I went to the meeting and proposed that like the change and the leaders of the team, they all agreed. that is forced to be implemented. So that's it.

Bart: Fantastic. Good. And really nice story that you yourself, you know, building your own KEP and that's possible as well. Great. So in terms of you personally, what are your next steps with this project, with any others that you're involved in? What are you going to be doing next?

Maksim: So I hope that I have more ideas what can be fixed in Kubernetes because I work with it a lot. So I believe that this year I will introduce some more changes, some more ideas to implement. Yeah, and I will also continue supporting Dex project. And like by taking the chance, we are searching for more maintainers for Dex. And like if you're interested, you can contact me. Yes, so that will be it. And I also plan to attend conferences in Europe. So if you're a KCD fellow, for example, probably we will meet. Yeah, that's it. And the plan for the work is just to continue growing, continue developing the internal platform, continue helping clients with their services, with solving their complicated problems. I hope we'll improve also in this field.

Bart: Sounds like a good plan. Is there anyone in, any people that you would like to give a shout out to who are instrumental in the KEP?

Maksim: I would like to give a shout out to Mo. He was all the way through, he was, I would say, like my mentor. I could ask him any questions and he answered always, always shared his ideas with me. So I'm glad that... I know such a great person. Yeah. So shout out to Mo. And if it's possible, I would also, I would like to give a shout out to Mark. Mark is the second core maintainer of Dex project. So Mark, if you see me on TV, if you hear me on radio, hello.

Bart: Very good. Thank you both. Thank you, Mark. Well done. Good. What's the best way for people to get in touch with you if they want to get involved in the project? Like you said, Dex, you're looking for maintainers. What's the best way to do it?

Maksim: LinkedIn, GitHub discussions, and probably email. By priority, this is the list.

Bart: Okay. All right. Well, Maksim, thank you so much for your time today. Really appreciate hearing from you and your experience, and best of luck in the next steps.

Maksim: yeah thank you so much for having me today