Bart Farrell: What happens when a Kubernetes service account token ends up in your production logs and it's valid for a year? In this episode of KubeFM, I'm joined by Vincent, a platform engineer at IPT in Switzerland, to unpack a real-world incident that exposes some of Kubernetes' most misunderstood defaults. We break down what's actually inside a service account in JWT, why audience scoping matters, how vault integrations can introduce hidden risk, and how legacy token behavior can quietly expand your blast radius. We also cover the evolution towards projected tokens and the TokenRequest API, and what platform teams should be doing today to reduce exposure. If you care about Kubernetes identity, authentication, and security trade-offs in production, this episode is definitely for you. This episode of KubeFM is sponsored by LearnKube. Since 2017, LearnKube has trained Kubernetes engineers all over the world with their courses, which are instructor-led, 60% practical, and 40% theoretical. They are given in-person and online to groups as well as individuals. and students have access to the course materials for the rest of their lives. For more information, go to learnkube.com. Now, let's get into the episode. Vincent, welcome to KubeFM. What three emerging Kubernetes tools are you keeping an eye on?
Vincent von Büren: So, my three technologies are closely related to the work I do every day. And the first one is one which is dear to my heart. It's CloudNativePG, just because I know the struggle of working with Postgres within the Kubernetes ecosystem. And there is a lot of, I think, interest in CloudNativePG because you have finally a running Postgres operator in a Cloud Native way. And it kind of takes away the stress from the whole Postgres related stuff. I mean, self-healing stuff, lifecycle, point-in-time restores and stuff like that. So I'm really interested in that technology. There's actually some colleagues who do some POCs with that technology and I'm looking forward to finally test it out, maybe in a few months or something like that. So that's, I think, the first one. The second one is also very close to our project. It's Atlantis. Not sure if you've heard about Atlantis. It basically allows you to run Terraform workflows when you create a pull request. So we use Terraform to configure certain external systems, like for example Vault, just to have some persistence of your configuration. Right now, most of the steps are manual. And if you want to have it in an automatic way, then you have to go, in our case, a bit crazy with Tekton pipelines. And I mean, I think Tekton pipelines are cool, but sometimes they're also quite hard to debug and do stuff with them. And especially like Terraform stuff isn't like something completely new. So in that sense, I'm really eager to try out Atlantis as well. Again, there are some colleagues of mine. I never do the fun stuff. Colleagues of mine doing some work with Atlantis and trying it out. And then the last one is more on a personal interest because we're providing some service as a service stack on the platform and our customers are developers and for developers, for additional developers, the whole Kubernetes role is still quite strange to them. So I'm looking at Crow, it's a resource orchestrator. which allows you to basically have an interface of different Kubernetes resources on top. So you can define multiple resources together, be it network policies, services, ingress, whatever, and have those in a resource graph definition, I think it's called, and pass that to the developer. So they don't have to care about all these different Kubernetes resources anymore and just deploy their application on a golden path. And I think This will be quite nice and interesting for our customers because it takes away this cognitive load of dealing with the Kubernetes world, which is not very close to them.
Bart Farrell: And Vincent, for people who don't know you, can you tell us about what you do and where you work?
Vincent von Büren: So I'm working for a company here based in Switzerland. It's called IPT, which is short for Innovation Process Technology. And I'm working there as a senior IT consultant. So my mom still thinks I'm doing slides, but that's actually not true. I'm a developer, well, platform engineer right now. And IPT is, as the name suggests, it's an IT consultancy. So we are focusing on the Swiss-German market, covering the banking, insurance, and public sector, and then covering the typical software engineering fields from cybersecurity, AI, but then obviously a lot of cloud, cloud-native topics as well as cybersecurity. And I'm now. Two and a half years working on a Cloud Native project with some fellow colleagues, and it's a lot of fun and quite an interesting journey so far.
Bart Farrell: Great. And tell us about how you got into Cloud Native in the first place.
Vincent von Büren: So my background is more, so I have quite a weird background. So I started with law and economics and I thought like being a lawyer would be quite a nice thing to do. And then I realized that it's quite boring. So sorry for all the lawyer listeners in your podcast. So I made the switch to a more technical background and I started up with electroengineering, which was also quite fun. And a lot of electroengineers, they end up in software engineering because hardware is hard. And also you can like work from everywhere, right? So I started up with more classical software engineering, backend engineering, a lot of front engineering, business-like aesthetic stuff. And eventually I ended up in Cloud Native because as an IT consultant, you have the possibility to try out new technological fields. And I got staffed on this project and I wanted to get my hands dirty on Kubernetes. So that's how I ended it up in the cloud native world.
Bart Farrell: And what were you before Cloud Native? I understand the legal background, but in terms of what kind of technologies were you working with prior to getting involved in this area?
Vincent von Büren: So I would say I did a lot of data engineering stuff and data science stuff with the typical stack, Keras, TensorFlow back then, and Python, obviously. And then I also had some fights with frontend, React mostly, or TypeScript. And between that, I did some additional scripting. But I mean, I always saw myself as a full stack engineer because I love to touch all these different fields and was eager to kind of learn new technologies, right? So I would say most of it front-end, most of it data engineering and data science.
Bart Farrell: And the Kubernetes ecosystem moves very quickly. How do you stay up to date?
Vincent von Büren: so i obviously use social medias and there's some great content contributors There is a ThePrimeagen obviously ranting about the whole world there's TechWorld with Nana there is a DevOps Toolkit from Viktor so there are many different channels which keep you up running. And besides that, I would say a lot of meetups as well. I mean, in Zurich, you have the Cloud Native Zurich. So shout out to them doing a great work, bringing the Cloud Native world together. And then, which is quite awesome at our workplace, we have a lot of kind of internal dev conferences. So every month, the whole company comes together and we have like workshops and talks about different technologies. And Cloud Native is obviously on the front there. and also hackathons and we call them tech beers. So it's basically coding and drinking beers. So I would say exchanging with my colleagues.
Bart Farrell: A winning combination for sure. I would also like to echo the shout out to Cloud Native of Zurich. I haven't been able to go to a meetup there. I would like to, but I've just seen it for years, time and time again. Very active community in terms of bringing people together, of consistently building great events and a lot of great interactions happening there. So yes, a massive, shout out because that is no easy task. A lot of patience in scheduling and planning. And so much, respect for the folks that are doing that out there. So Vincent, as part of our monthly content discovery, we found an article that you wrote titled, that time I found a service account token in my log files. So we want to dig into this a little bit more deeply to get to hear about your experience, the choices you made and recommendations you would make to folks out there that might find themselves in a similar situation. So here's a study. It's a quiet Tuesday. You're cleaning up an old Helm chart, scrolling through some go code and then you spot it. A debug statement logging a Kubernetes service account token. Still in production. What went through your mind in that moment?
Vincent von Büren: Oh my gosh. Just thinking about it makes my heart race again. I mean, obviously my stomach turned because I knew that the implications of it, this could potentially be quite a serious security incident. And I knew, I mean, I was refactoring this Helm chart. I knew that this Helm chart was rather old and probably there was no like modern way of injecting tokens in place. So I was using my jwt-cli tool to kind of decode this JWT token and voila It was a no audience set of normal specific audience set and the validity was one year. So quite a quite a cool combo. And after that my mindset shifted a bit. I mean I was then like calling my colleagues and we're like, okay, what should we do? So we basically followed incident response, we stopped the further leakage, we looked at the blast radius, okay, what's actually impacted, right? We changed the credentials and we updated the workload to no longer publish this credential into production. And this was, I think, the first reaction. And then after that, props to my team. When we talked to different people and understood how serious that is, we had a good cause analysis and we analyzed it and team together and make sure that it hopefully no longer happens in the future.
Bart Farrell: For listeners that are out there who might not work with Kubernetes daily, a service account token floating around in logs might not sound alarming, but you call this a time bomb. What can an attacker actually do if they get their hands on one of these tokens?
Vincent von Büren: To understand a bit that, I mean, in the Kubernetes world, you need identity, right? And this is dealt with service accounts or service account tokens who are then like mounted into the file system of pods. And depending on and they use JWT for that. Depending on your JWT token, you can do all types of stuff. And as mentioned, it has two bad things. First of all, there was no real audience set, which means that with this token, an attacker could basically impersonate this workload and interact within the cluster based on URL-based access control. So you can do a lot of dangerous stuff. You can potentially interact with the Kubernetes API server to list or read any secrets. You can do some execution commands. So it was really a time bomb. And on top of that, with the validity of one year and being an old legacy service account token, people who have access to your token can use it for a long time, right? So it's basically a live credential. Like you said, these tokens are essentially bearer credentials. Whoever holds them is that identity.
Bart Farrell: What information is actually embedded inside a service account token and what did you see when you decoded it?
Vincent von Büren: That's a great question. So I think it's a great recap as always for myself, because I always read that again. So a JWT token consists of a header, payload, and then a signature. In this case, the payload is the interesting stuff because it contains all the interesting claims between two parties. And the first thing is your issuer. Your issuer is typically, in that case, the Kubernetes cluster, the Kube API server, which issues this token for you. And then the heart, I would say, is the subject. It's the most important field and it basically tells for whom this token is for. And because it's a service account token, you typically have your service account name and also the namespace where your service account lives in there. And then you have your audience. And the audience is basically who is this token intended for and because it uses the default Kubernetes address, internal address, everything which basically trusts the internal Kubernetes API also trusts this token. And also every external system which is tightly connected to Kubernetes also then maybe trusts this token. And having that, having this audience set to something like that is dangerous because you can have some replay attacks and people can use this token to do any malicious stuff. within the Kubernetes cluster or with the external systems. And then you have some additional fields like expiry, which is in this case set to one year, and the issued at, right, which is just a timestamp.
Bart Farrell: And the point about the one-year expiry stands out because if someone accidentally logs this token today, it's valid until next year. And you mentioned it gets mounted into every pod by default, even pods that never talk to the API server. So how did Kubernetes end up with these defaults?
Vincent von Büren: because Kubernetes is great. Now, jokes aside, I think it has to do with the historic context of Kubernetes. I think in the early days, the design goal was more on usability and automation, and everything within the cluster was basically considered trusted. And pods needed maybe access to the internal Kubernetes API to, I don't know, handle some controller stuff, do some legal action, service discovery, whatever. Then over time, because this isn't the issue which is very new, Kubernetes realized that the community learned that they have to take them more seriously, following the principles of least privilege, evolving basically also Kubernetes in the sense that nowadays there are a lot of features that make sure that this isn't the case anymore. So pods shouldn't have API access right off the buck. And there's, for example, token request API, which again, nowadays allows you to handle service account tokens a bit more elegantly and more secure. So I think it was really like The shift from usability and automation in the beginning to then more to security by default approach nowadays.
Bart Farrell: There is another property in that JWT that the audience claim. The token you found had the default Kubernetes API server as its audience. Why does that matter?
Vincent von Büren: I mean, I compared that in my article with like a concert, a Coldplay concert. So imagine you have a ticket for it. concert happening this march in Zurich. So your audience is like this march and the stadium in Zurich where the concert will happen. And maybe there's another concert in a year in London, something like that. And if you don't have specifically an audience set, then you could go with your tickets to every concert, right? So the audience claim is really important, and you make sure that only the system which is intended for can use this token validated. And if you use the internal Kubernetes API address, then every system which trusts this address can basically be attackable. So it just increases the blast radius massively.
Bart Farrell: And so we have tokens that live for a year, have no audience restrictions, and get mounted everywhere. Three strikes. But I want to go back to that Tuesday morning where we started, you found this line of code logging a token in plain text. How does something like that even ship?
Vincent von Büren: I think there was a combination of different stuff together. I mean, back then we had a lot of time pressure, delivery pressure. So I think it was just a human-like fuck up, which just happens to every one of us over time. I mean, I wouldn't be the first, right? And then the second thing is service account tokens per se. they don't look so dangerous. So they're just mounted into your container file path and maybe from a dev perspective you just think it's just some token and you misunderstand the seriousness of this token and how this token can be used in a dangerous way. And then also back then we did a switch in our CI CD pipeline so we opted for a pipeline as code approach with of Tekton. So I think there was also like some churn going on there and the combination of that time pressure, human fuck up, and CI/CD guardrails not in place made it happen that this token was leaked.
Bart Farrell: And this is kind of what got extra attention. That debug statement was there because someone was trying to get Vault working. The system designed to keep secrets safe. If the Lord of the Rings fans out there, keep it secret, keep it safe. And in the process of integrating with it, they logged the very credential that grants access. There's a little bit of dark irony here. How do service account tokens and Vault actually fit together? And why does this integration trip people up?
Vincent von Büren: There is for sure, unfortunately, some irony to that. And I mean, I think the problem was that Vault per se was used, or this token was used to bootstrap as a bootstrap credential, right? So it was used to authenticate against Vault and then you get a token from Vault, a short-lived one, to do whatever you want to do with the Vault. And I think this integration step isn't that trivial as all because you have to configure policies, you have to configure audiences, you have to make sure that the bindings are correct. And this is a step which can be quite unnerving and you have to make sure that the authentication works quite nicely. And having that setting, I guess, it was convenient to do. some debugging and see if the authentication actually works and the policies are in place especially because we do everything again with Terraform and then running Terraform and having pull requests and stuff like that it just takes time right so I think that's just one factor and then the other factor is I always believe that having Vault in place gives you this weird feeling of security because we think everything which is secret related is happening in Vault. And then you forget about it that service account tokens itself are quite sensitive. And if they leak, you have some potential damage happening there. So having Vault in place obviously doesn't remove the sense of you have to protect your secrets. And it shows that, Kubernetes identities and those secrets must be protected by themselves.
Bart Farrell: Vault offers two ways to validate these tokens. The Kubernetes auth method and the JWT auth method. They sound similar, but have different trade-offs. When would you choose one over the other?
Vincent von Büren: So there is this quite neat feature that you have, Kubernetes auth method, which means that Vault basically communicates with the Kubernetes API server. I think they talk with the Token Review API. they validate if this token is actually coming from Kubernetes or not, and then based on that, they exchange the token and you can do some crazy stuff. So there's some quite tight coupling, which made it quite easy as well to work with it and to set it up. What you need for that though is you need network access on both sides, and you have to trust the Kubernetes API. So if that happens within the cluster, If you use, for example, a vault within a cluster to do some kind of access management, secrets management, it's quite the way to go. And then the other way is, in general, like JWT authentication. And there you don't have such a tight coupling. I mean, you basically have vaults then validating the token, the JWT token, against OIDC and see if it works and if the token is valid. This allows a bit more of a cross-platform approach. So if you want to, for example, use Vault to do secrets management, access management, cross-platforms, it's a way to go. But obviously, it's harder because you have to make sure that OpenID works and you have to make sure that Vault is able to connect.
Bart Farrell: So we've got long-lived tokens, no audience scoping, automatic mounting everywhere, and a vault integration that's tricky enough that people resort to logging credentials to debug it. Is there a way out of this? And has Kubernetes evolved to address any of these problems?
Vincent von Büren: Luckily, yes. It's actually, they evolved a lot. So one thing which was a big improvement was the introduction to token request API. So this idea of having like these static legacy service account token isn't like something which we have nowadays in the newest versions. of Kubernetes. Nowadays, how it works is if you do actually mount or you want to mount some kind of service account token into your pot, then I think kubelet communicates with the token request API and then you get the projected volume, a service account token injected into your pot. And, which is quite nice, you have to specify the audience and also the time to live. Now you have a token which is maybe valid only minutes or hours, which obviously, again, make the blast radius quite small. And you have the audience scoping, which then limits the risk of some kind of replay attacks. And on top of that, having the token request API, you can also, as mentioned, remove the need of having this automatic mounting of tokens and pods, because not every pod needs access to the Kubernetes API server.
Bart Farrell: Short-lived, audience-bound, auto-rotating. If a projected token leaks into a log file the same way that legacy token did, how is the situation different?
Vincent von Büren: I mean it's still not cool, it's still a mess up, but obviously again the blast radius isn't that great. So I wouldn't say it's a huge deal, it's a tiny little blip, because you have way a shorter lifetime defined on your time to live, maybe minutes, maybe hours, then you also have automatic rotation of your of your Kubernetes tokens. So it can be that Kubernetes refreshes the token and the leaked token is no longer valid. And hopefully you have audience restrictions in place so you don't have this issue of anymore that having the default Kubernetes internal address leaked to everywhere and you can use it.
Bart Farrell: Projected tokens haven't been available since Kubernetes 1.20. That's some years ago. If they're clearly better, Why isn't everyone using them? What's holding teams back?
Vincent von Büren: So I was looking, it was back 2020, so really some years ago. I mean, I think it's an example of like, you have all the secure features, but they're not yet really in place because of various reasons, right? I think one issue is for sure that if you touch any kind of identity and authentication on platforms, you always get, or at least I get always a bit sweaty hands because it touches many different systems and you don't want to up there. Maybe you have some backward compatibility issues that you have some old legacy controllers who still use the old service account tokens. You have some kind of cross-system coordination with external systems, right? CI/CD pipelines, Hashicorp Vault, where you have to configure them at the same time and this takes time and planning. And then, I mean, we're all humans, And so it's also, I think something between awareness and priority, right? I mean, when stuff gets leaked, you always have high priorities. So if you want to get stuff done, always leak your stuff because then you really can do stuff. But I mean, maybe the mental mindset is just these tokens work and until then they just stay invisible until the incident happens, right? And last but not least, I believe also some operational fear, right? I mean, if authentication breaks and you have some pods, some workloads who no longer work and outages, I mean, it just feels very risky. And if you don't have proper monitoring and observability in place, you don't want to do any. big changes there.
Bart Farrell: For platform teams that are out there listening who suspect their clusters still run with default tokens, where should they start?
Vincent von Büren: So as mentioned previously, not every pod or every workload needs access to the Kubernetes API. So I would first identify those and figure out, okay, do they actually need that token? And then disable for them this kind of automatic token injection from the TokenRequest API. And then if you have some workloads who actually need access to them, who need identity and need access to the Kubernetes API, make sure that you use these injected service account tokens with accordingly set time to live and accordingly set audience. So you have the scoping in place. And then finally, guardrails, right? You want to have your CI CD in place that if something gets leaked, you're, your pipelines fail and you get alerted, right? So prevention is always key.
Bart Farrell: You built a hands-on lab so people can experiment with this themselves. What's in the repo? And what will someone learn by running through it?
Vincent von Büren: So again, I love to play around, right? And nowadays it's so nice to interact. I mean, I use in my hands-on a kind, which is KinD and Docker. And I set up a little cluster and also a vault instance in dev mode. So it's also easy training to act with Vault and basically can simulate the different authentication methods. So you can play around with Kubernetes authentication method, you can play around with JWT authentication method, and it also shows you how these tokens work. You can decode them, you can actually debug them in a safe way if you want to do that. And also have an example in there where the audience isn't set correctly. So you also have an understanding of how this scoping actually works. So for me, it was It was more for me actually in the beginning because I thought like, okay, I really want to understand that in detail. And then obviously I want to share with the community because why not help people understand that stuff as well. Vincent, what's next for you? I mean, parenthood. My wife and I were expecting the birth of our first baby girl. So I will unfortunately, take some time off in the cloud native world. But my goal is still to attend the Swiss Cloud Native Days in September happening in Bern. But let's see if my daughter and my wife will agree with me and let me go.
Bart Farrell: Okay. Congratulations on that project. That's absolutely I mean, if writing articles is challenging, I think you're going to be ready for some other ones. And if people want to get in touch with you in the coming months, once you're a little bit more established and can get more than two hours of sleep per night, what's the best way to do that?
Vincent von Büren: Hey, just reach out in LinkedIn. And let's chat. I'm open. super keen and hopefully also maybe next year at KubeCon or maybe in September in Bern.
Bart Farrell: Perfect. Vincent, thanks so much for your time today. Hope our paths cross again in the future. Take care.
Vincent von Büren: Thank you so much, Bart.
