Kubernetes base64 secrets are fine

Bart: There's an expression that some like to use that says, the best defense is a good offense. When it comes to Kubernetes, when it comes to secret management, when it comes to threat modeling, thinking, anticipating how our systems can be attacked, how weaknesses and vulnerabilities can be exploited, we always need to be a step ahead of it. In order to do so, I got a chance to talk to Matt, who's been thinking about this a lot, has some controversial opinions about Kubernetes secrets, and I'm pretty sure you're going to enjoy this episode. If you too are thinking about how your system might be in trouble, how it might be in danger, and based on that, how you should properly approach the topic of threat. Check it out. All right, Mac, welcome to KubeFM. Just want to jump in with the first question. If you had a brand new Kubernetes cluster, which three tools would you install?

Mac: I think it's a trick question because you'd only install one tool which is Flux and then Flux would install the other tools. But yeah, I definitely really like Flux where you can just git push and it will automatically deploy stuff to your cluster. I think that's really nice, especially when collaborating with coworkers. And then next would be probably kube-prometheus-stack. just in terms of sheer monetary value because of how expensive monitoring solutions are out there. Like you probably have to put a second mortgage on your house if you use something other than kube-prometheus-stack, which includes like Prometheus Operator and Alert Manager and Prometheus itself, and a bunch of pre-built Grafana graphs and alerts and things which is really nice. for keeping your cluster running. And then the third was a bit of a toss up for me. I ended up going with Gatekeeper, open policy agent Gatekeeper. It is, I think in, Helm charts these days are getting so complicated that when you go to install them, like, there's no way. You're kind of just read through every single line of them. And so at least with Gatekeeper, I can ensure that there's no like privileged pods being created or, and you can enforce other kinds of more, more creative policies like blocking ingresses with a wildcard hostname, which is something that I think more people should be doing because you can kind of steal traffic from one of your co-workers if you do that. But yeah, that's what I install.

Bart: Fantastic. You mentioned Flux. We had a previous guest, Mathias, who also talked about the fact that if he were to install three tools that Flux would be one, and that he was explaining a little bit about what he saw in terms of trade-offs between Flux and Argo CD. In your case, what are the benefits of Flux that outweigh what Argo has to offer?

Mac: Well, they both, do this unfortunate thing which I don't like, which is that they kind of demand full access to your cluster. And I like to make sure that the operators I'm installing don't have that access. I think the only difference is that FluxCD, if you've made it right, you can get it to just operate in one single namespace. And I believe ArgoCD, maybe it has that support now, but it much more prefers and expects a cluster admin role assigned to it. And then it has its own authentication thing. And plus it has additional stuff, like it has the UI and whatnot. and I prefer much more minimal. Like if I'm gonna give something, elevated privileges to my cluster, I want it to be really minimal.

Bart: Now that we got that out of the way, can you tell us a little bit more about who you are, what you do and where you work?

Mac: Sure, yeah. So my name's Mac, and I guess I'm just your friendly neighborhood Kubernetes cluster administrator. I actually recently changed jobs, but generally what my role is at all my past and current jobs as being the person in the room who knows the most about Kubernetes and then we have an application that runs on Kubernetes so naturally I stay very busy. But I also just do general DevOps work like setting up CI/CD lines and but in addition to all that stuff I have a pretty busy day. of a side interest in security. Did a lot of security compliance at a previous job and I didn't find it as torturous as probably some other people do. So, and as a result I kind of have some some opinions about security and I've been doing a lot of reading about it. And so, yeah, that's another big thing that I do.

Bart: Not often that we find, like you said, a sort of security enthusiast who doesn't find these elements torturous, but we can get at that a little bit later. Tell me about what you were doing before you got into cloud native, what the process was like getting into cloud native technologies and eventually learning Kubernetes.

Mac: Sure. So it was actually my very first internship. I was only a sophomore in college. That included Kubernetes work, which is actually a funny story because the app that we were building was a really simple like Python web app that talked to a database and so we absolutely did not need Kubernetes on top of that. But of course I was just an intern and I didn't know any better and some of my other co-workers were arguing for it and so I must, I guess I have to thank them now though because it was a very good learning experience and then I could use that experience to go get my next job and my next job.

Bart: And in terms of, you know, if you had to give your previous self advice about learning Kubernetes, which resources to focus on, what would be helpful, anything to not do, what kind of recommendations or tips would you give to Mac when you were starting out?

Mac: Yeah. Yeah, one thing about like my initial internships where I had to learn Kubernetes was that they were very independent. And that's kind of a blessing and a curse where you don't have a whole lot of guidance. You get to, you know, learn things the hard way, but it takes a lot of trial and error. And so that's why I think that mentorship is kind of really important, because a mentor can at least guide you in the right direction to do all the trial and error and save you a lot of time and make sure that you learn the right things and you don't go off so in some wrong direction. And as a result of that, I've been trying to do mentoring kind of activities myself, like volunteering at hackathons and things like that.

Bart: That's great. Wonderful form to pay it forward. Giving back to others also helps you consolidate your knowledge when you have to share it with somebody else. And also playing a good role in the broader community. In terms of what we wanted to focus on today, you wrote an article about plain Kubernetes secrets are fine. Can you tell us first a little bit about what Kubernetes secrets are? And just so we have a decent starting point and then we can take it further.

Mac: Sure, so Kubernetes secrets are a Kubernetes resource, just like all the other resources like pods and deployments. nodes, and all the resources is like a YAML, essentially a YAML file with a specific structure gets stored in etcd, the database that backs your Kubernetes cluster. So yeah, secrets are no different, they're kind of a YAML file that gets stored in that database. There's one catch though, which is that the secrets are generally represented as base64 encoded. And this can kind of trip some people up because they think that this was maybe a mistake, that the secrets were supposed to be encrypted. But really the Base64 encoding is just used so that you could store binary data as a secret. Like you could have a secret JPEG or you could have a secret binary database backup, for instance. But yeah, that's essentially what they are.

Bart: With that in mind, something we do here a fair amount, whether we're talking about encoding or encrypting, the idea of a threat model, and I'm waiting for the fashion world to appropriate that term and make it their own, but what is a threat model and how can it help people that are working with Kubernetes to encrypt their secrets?

Mac: Right, so a threat model is just a way of thinking about the kinds of attacks that could befall your application and the kinds of defenses that you could stand up to prevent those things. And it may sound like that's intuitive, but actually a lot of people kind of skip that step because it's pretty easy to get lost in doing like compliance box checking and not thinking about like, well, actually, is it, is it physically possible for someone to get physical access or is it possible for an attacker to break this kind of encryption? Do I need to care about that? So yeah, in relation to secrets then, using a threat model can tell you how additionally complicated you should make your secrets handling to resist against particular kinds of attacks. But what I argue in my blog post is that under most circumstances, just the plain Kubernetes secrets that come out of the box generally are pretty good against the kind of impossible attacks that you might face of someone stealing your secrets.

Bart: And you mentioned this notion of simplicity, making things easy. Without getting too overwhelmed with lots of details, diagrams, graphs, how can someone define their own thread model?

Mac: Yeah. I definitely, threat modeling is kind of a whole field of study, like it has special diagrams and methods. I think Microsoft has their own method for it. There's even people who apply formal methods to threat modeling, but I say forget all that and just do something that makes sense to you. So just write it down. It just takes two easy steps, I say. First, think like an attacker. Put yourselves in the shoes of an attacker and come up with some possible ways that they could exploit your code or whatever. And then the second step is just come up with possible defenses, ways to stop those particular attacks. And once you have that, you can make it much more fine-grained or you can just keep it high-level, but it can be a really good guide for planning. future changes or architectural changes to your code.

Bart: And let's take this example, let's take, you know, Kubernetes secrets as an example. What would be the steps? How would you, you know, could you guide us through the process of building a threat model?

Mac: Sure, so yeah, in my post, I, again, I did it pretty free-for. I first started by kind of defining secrets and I made it a bit more concrete by referring to a particular kind of secret. So, like a database password is probably a pretty common type of secret. And then from there, I defined what it means to... have a security failure and that would be someone, of course, stealing the secret, but importantly, stealing the secret and then using it to log into the database and grab data. That's a real security failure. And once that's defined, you can kind of get into that thinking like an attacker. And so I had defined five different attacks or things that could befall our secrets and Again, it was just the most I could come up with. And would you like me to go through those five things?

Bart: Absolutely, if you could. And then for each one also, if you want to add, you know, what would be preventative measures that could be taken to stop those attacks from happening?

Mac: Sure. Yeah, so the first one I came up with was like a malicious process running on the same node as your application. Just scanning memory or reading the value from /proc if you're using environment variables. Yeah, that's a pretty difficult, if not impossible, thing to avoid in the world of containers where you have containers of varying security levels running on the same physical or virtual machine. and then the next one was root access to a control plane node. If you have root access to a control plane node, someone could access etcd and just directly read the data out of there, your secret data, and then use it to attack your application. This can kind of just be stopped with general hardening, but again, there's always a risk that that could happen. Next is root access to a worker node. So worker nodes obviously don't have access to etcd directly but what you can do is you can grab credentials from kubelet which runs on every node and then use that to get access to the secrets. And or if you happen to be on the same node that the application is running on again you can read it straight from /proc or from memory. And again, this is something where if you have different... If you have privileged applications running in your cluster, it's kind of very difficult to avoid. Fourth attack is physical access to the server. Like if someone breaks into an Amazon data center and grabs your data. Again, the best way to prevent this is just good physical security. But if someone does get physical access to your server, there's kind of nothing you can do because there are even tools out there. It becomes tense. where you can plug something into a PCI slot of a running server and it can extract the RAM and the person can put it in their pocket and walk out the door. And of course all secrets have to exist in RAM in some way for applications to use them to log into a database for instance. The fifth attack is kind of just a catch-all which I just called future unexpected attacks. If anyone was around, I think in 2019 in this space, there was a, that was I believe the most recent container breakout on our ability that affected maybe containerd or something. And that could happen tomorrow, knock on wood. But, um... So what this catch-all attack lets us do is it lets us prioritize simpler, easier to reason about defenses, because if you add more complexity, you're adding more attack surface, and you're adding more... things that can go wrong at some point in the future.

Bart: You identify, you know, these five. Once getting through those five, should someone stop there and say, all right, I'm good for now? Or should it be an ongoing practice to add new scenarios in the future?

Mac: I'd say it's definitely should be an ongoing practice because in this world of having to think like an attacker, attackers are not going to give up as soon as they run out of ideas, they're going to come up with new ones. And so maybe five years ago no one would have thought to, for instance, protect against timing-based side-channel attacks like Spectre and Meltdown. but now we have to think about those things. So that again, tomorrow, so a new one of these could come out. And so I'd say, you should always be updating your threat model. I took like a cheesy security course at a previous company and the motto was "security is a journey, not a destination". And I think that's a pretty good way to sum it up.

Bart: So with that in mind, it sounds like I can just skip Kubernetes secrets altogether, right?

Mac: I certainly wouldn't want to go back to the dark ages before Kubernetes secrets existed when there were a bunch of competing standards or no standard at all. At least now with secrets, we can kind of have a platform to interoperate. And there are many alternatives secrets that are actually built on top of secrets itself, like, you know, Bitnami sealed secrets are one that I talk about. They're not really an alternative to secrets per se because how they work is that you can commit an encrypted file to like your Git repo and then when you apply it to the cluster there's an operator that will decrypt the Bitnami sealed secret and convert it into a plain secret, Kubernetes secret. In my article I go through all these alternatives. I kind of go through each of the five attacks that I had listed out and in the case of the non-esealed secrets, they don't actually protect you against these kind of any additional attacks than playing Kubernetes secrets. And that's okay because it's kind of it has a different goal. It makes it the goal of sealed secrets is to make it easier for you to collaborate and commit secrets to your git repo. But there are some other alternatives as well that do try to make themselves out to be more secure than plain-and-tidy secrets. I guess I could talk about those.

Bart: Yeah. So, you know, in terms of the alternatives, yeah, in terms of the alternatives that are out there, I'm sure some of our listeners, you know, might be thinking, well, you know, I use Vault. This problem is already solved for me. What would you say to that?

Mac: Yeah. Um, I'm. I'm not, I would never advocate for tearing out an existing working secrets infrastructure because I know how complicated those are to stand up. And so if you say that Vault's working for you, I believe you. I think that under the right circumstances it can. What I argue in my post though is that I've seen a lot of Vault instances that aren't properly configured. Like for instance, one of the big benefits of Vault is that it has Shamir secret sharing which allows you to encrypt the database using multiple keys that you can spread amongst people so that no one person can unlock the database. However, what people commonly do is they just disable that for a different feature called auto-unsealing, which is you give the keys to your database to a cloud provider, and then what that allows you to do is if the database, if fault restarts, it can automatically unseal itself, which kind of defeats the purpose of the sealing process because if an attacker were to get root access to your server where Vault's running, they could just grab the cloud credentials that are used to get the unsealing keys, and then unseal it, and then grab all the data. So that's one thing. And then another is that another way Vault's instances are kind of misconfigured is they have this rich policy language that lets you define who can have access to what. secrets, but I generally see people ignoring that and just using Vault as kind of a key value store. But yeah, so there are many ways in which it can be misconfigured, and this is kind of just a par for the course for a product that is kind of quite a bit more complicated than something like Kubernetes secrets. And so yeah, if people say that they have a good working Vault instance, then I say that that's really true because I have seen that work out. I've had, like if you have like HSM backed Vault instances and they're in an HA configuration and you've got a team of people who are dedicated to doing that or you pay HashiCorp for this kind of managed service, then it can be a good thing. But going back to threat models, it's important to actually do a threat model so that you can know whether.... building like a whole team to run vault is something that will actually give you kind of a return on investment in terms of security or whether it's maybe time is better spent doing something else.

Bart: A good friend of mine from Middle Earth Gandalf once said, keep it secret, keep it safe. From the things you've been mentioning here though, it sounds like you're never safe no matter what you do. Is that true or what's your advice here?

Mac: Yeah, well, you may joke, but I really do think that's true. Like with the new security breach in the news every week, I think that, yeah, really, you are never safe. My advice would just be, yeah, just do threat modeling, because one side benefit of threat modeling is that it helps you prioritize the work to do. because you can look at the likelihood of a particular kind of attack and you can look at how complicated it would be to implement a particular kind of defense and you can implement the easy to do things that solve the most amount of attacks. And yeah, when you're faced with a giant list of security tickets, having a threat model means that you can prioritize them easier and just knock them out. And then you can do the tail end of the less critical security things later.

Bart: With that in mind, you know, it sounds like this has been quite a journey for you. And it's also interesting because while we're talking about, you know, the context of cloud native technologies and Kubernetes, this approach of looking for vulnerabilities, anticipating potential attacks, sounds like something that can be used in lots of different aspects of life and something that's also used in business, you know, a framework for what's SWOT analysis, the acronym SWOT for strengths, weaknesses, opportunities and threats that can be used for a business or a piece of software. How did you learn all of this? What was your process? And for folks that are interested, what resources would you recommend?

Mac: Sure, so I have a lot to thank for a security class that I took as an elective in college. And I honestly believe it was one of the most relevant classes to my career, even more so than my other computer science classes because I got a computer science degree. And I guess I will say, first of all, that I think maybe security courses should be included in the core curriculum and not these electives, considering how important it is and how kind of everybody has a role to play in the modern security world where... everything is a potential weakness that can be exploited. Well, the first thing I would say is if you can find opportunities to learn on the job, that's probably a really good one because your time is already... we spend so much time at work and it's a lot of fun. leveling up your security knowledge is directly beneficial to your employer in many ways. And that was some of how I got started. When I worked at Cisco I had to take a kind of extensive security training course because when you're working on the routers that power the modern internet you got to make sure that those are a little bit more secure than your average Python web app. But as far as particular resources, just kind of reading a lot on the internet, but I particularly like a book called Security Chaos Engineering by Kelly Shortridge. There's a, I think there's a sea change happening in the security world where kind of people are realizing that the older model of strict, rigid compliance is really not working, as you can tell by the number of breaches in the news. And Kelly's book is very much tells you to be more practical and flexible and to do things like threat modeling when like actually think about the attacks that might affect you rather than the things that some paper pusher might think that you should do instead. I would definitely recommend that book.

Bart: And also just embracing, like you said, creative, trying to think as an attacker thinks, you know, putting yourself in their position and the logic they'll be using when we first came across your, your blog post, we were, you know, a little bit surprised, uh, in terms of it being provocative, what was the reaction from the community to what you wrote?

Mac: Yeah, surprisingly positive because I was expecting a lot more pushback because I wrote the article as basically a response to other writings that I had seen online just saying that Kubernetes secrets are just flat out insecure and you should never use them. You should use these alternatives instead. So I was expecting to ruffle some feathers with it, but it seemed like it was mostly positive. My theory is that I think once you've had to stand up like secrets management infrastructure as like a DevOps person and you've seen how complicated it is, maybe my ranting and this blog post is something that appeals to you or you find it relatable.

Bart: That's wonderful. And giving people something that yeah, they can relate to because of personal experience is one of the best things out there because until you've gone through it, it's pretty difficult to really understand what goes behind it and the kind of. difficulties you'll be facing and challenges you'll be tackling. Now, will you continue to rant or share hot takes around security?

Mac: Oh, definitely. Yeah. I've already got two more that I published since the one on secrets. So there's some things to catch up on there. But at current company, I'm going to start doing a SOAP 2 compliance work. So I'm sure that'll enrage me enough to write another one as well. But so yes, that's a definite yes.

Bart: So if people want to, you know, share their thoughts or experiences or perhaps, you know, respectfully disagree with your hot takes around security, what's the best way for them to get in touch with you?

Mac: Well, I'd say don't. Don't get in touch with me. But you can find me hanging around the Kubernetes Slack channel sometimes if you bump into me. But if you do disagree, I would encourage you to write and publish your own version. because like there are hundreds of people every day working for companies trying to figure out how to, you know, be secure and implement their own secrets architecture and the only thing they have to look at is the stuff that's online. And so if you disagree I would encourage you publish your own thing. You might in a long form so that it's, you know, easily digestible and the person can make their own judgment and Hopefully if we all do this, maybe we'll make the world a more secure place. Because we need it for sure.

Bart: Certainly agree with that. Like you said, getting those ideas out there, questioning how we're doing things, is the status quo good enough? Like you said, if it just paper pushing when it comes to compliance, not using that creativity to think about how somebody might exploit a weakness, whether it's walking into a data center or other things that can be taken advantage of. I think there's a really good points to have. And that all of us, whether we like it or not, are security stakeholders nowadays. So, Mac, really appreciate your work. Thank you for sharing your knowledge with us today. And looking forward to future posts of yours and crossing paths in the cloud-native ecosystem.

Mac: Pleasure. Thanks so much.

Bart: Cheers.