Abusing Distroless containers: shell commands on shell-less containers

Bart: In this episode of KubeFM, we got a chance to speak to Harsha, who sheds light on some emerging Kubernetes tools, the intriguing concept of Distroless container images, and why container security should be at the forefront of every developer's mind. He and I discussed the benefits of using Distroless images, like their smaller size and reduced attack surface, while also tackling the myth that they're somehow inherently secure. Security at the end of the day is a container's journey, and Harsha guides us through the best practices, from selecting base images to managing dependencies and fortifying every layer of your infrastructure. In addition to that, we also explore the supply chain as a potential attack vector, the importance of signing artifacts, and validating sources to keep your environment safe. And as a bonus, Harsha will share what drives him to share his knowledge, along with a sneak peek into his upcoming projects, including a new podcast and a cutting-edge tool for simulating multistage attacks in cloud-native environments. This episode of KubeFM is sponsored by Learnk8s. How do you choose the best instance type for your Kubernetes cluster? Should you use few and very large instances, or many small ones? When using an 8GB 2V CPU instance, are all the memory and CPU available to pods? Is running the same Kubernetes node in AWS cheaper or more expensive than in Azure and GCP? The Kubernetes Instance Calculator answers those questions and a lot more. The calculator is a free tool that lets you estimate costs for your workloads based on requests and instance sizes. Explore instance overcommitment and efficiency. Identify over- and underspending by modeling error rates on your actual memory and CPU usage. Compare instances between different cloud providers. It's an easy way to explore cost and efficiency before writing any line of code. You can find the link to the calculator in the comments. Alright Harsha, first things first. Which three emerging Kubernetes tools are you keeping an eye on?

Harsha: Alright, I don't have any specific tools in mind but maybe I could think of Kyverno, if it is pronounced right, or Kyverno for that matter. That's one good tool I came across. I mean there are regular tools like kubeaudit, kubebench, kubearmor, and kubescape to basically access your clusters and find out what are all the possible attack paths in the clusters. So, Kyverno specifically because it is not just like an admission controller which uses validating and mutating webhooks but it can also clean up resources. For example, think of a scenario where I directly spun up a resource by adding something into etcd which is like an API server bypass, an admission controller bypass. In that case, I can clean up resources which are not violating a policy I had written. Having a policy also helps me clean up resources. So, I think that's a good tool. This is what I can think of.

Bart: Very good. With that in mind, for people that don't know who you are, can you explain who you are? What do you do? Where do you work?

Harsha: All right. Myself, Arshak Koushik. and currently I'm a security researcher at Palo Alto Networks, mostly around CNAPP. I started as a security engineer at a startup, and then I was doing Cloud-native security for a while, and then I moved to Disney Plus Hotstar. There also it's more or less the same thing, infrastructure security Cloud and Cloud-native. Post that, I joined Palo Alto. It's been six months, so this is a brief of what I do. How did you get into Cloud-native? What was that process like? I would say it's all by accident. I think the best things happen to us all by accident. I never planned that I'd get into infrastructure security as such, but my career started with networking, the typical CCNA, CCNP routing and switching stuff before I joined my college. After that, I got introduced to how the internet worked, which fascinated me. Then I got into the infrastructure security world. It started with Cloud, it started with AWS. Then I learned Docker, K8s, and other stuff like Linux, typical SOC, all these things. Post that, I was exposed to CNAPs. I have evaluated and was in the POCs and basically ordered a lot of calls and gift port. I've used three CNAPs to date. That's how all my work was around CNAPs.

Bart: And the Kubernetes ecosystem moves very quickly. What do you do to stay on top of all the things that are going on? Which sources and resources are the best for you?

Harsha: I think mostly any random newsletters. Some of the security stuff, basically client gibbler or somebody just keeps posting on Twitter. There are a lot of Twitter accounts. I don't recall them, but there are a lot of sources on Twitter to know what's happening. Apart from that, I think it's forums. Any of the forums, somebody keeps asking, "Hey, there's this new tool out there. What's the capability?" Somebody had already tried it. So what does it do? So then I just go and take a look at it. Mostly these.

Bart: If you could go back in time and give one career tip to your previous self, what would it be?

Harsha: One career tip to myself? Maybe something like, just go out and write, talk about everything you know, without always having something stopping you because you might think, hey, maybe this stuff is not that interesting. That's something which always stopped me two years back from writing or talking, but later I realized that what I know right now is something that younger folks actually want to know. Let's say you figured it out in the last two or three years, but they are in the state where you were in the last two or three years. The best way is to just write down things and talk about things without anything stopping you because we keep learning a lot of things in the security world day in and day out. For you to even look back and see what you've learned and do a validation, it takes something like writing or talking about it. That's a great reminder for you to see what you've learned. So in that sense, yeah, solid advice. Now for the topic of conversation today in our monthly content discovery, we found an article that you wrote titled "Abusing a Distroless Container" based on your experience with a real incident. The questions that we want to ask you will take this topic a little bit further. The article that you wrote revolves around Distroless containers. Taking a step back before diving into the episode's crux, if someone is a software developer who uses Docker build to build Docker images, what are Distroless images and why should anybody care about them? I've seen a video of this setting. There was a Defcon talk and also somebody wrote on Form 3, Daniel or someone. So I was just checking in my inferred Distroless containers, which have the same OpenSSL. And when I went and checked, there was something. So I just went and saw what I could do with this and then thought I could write it down too. The whole myth around Distroless is that everybody was going behind Distroless containers because of their size and slim attack surface. That was the whole hype about Distroless. The thing they call Distroless and what keeps it different from a distribution is just a few things. A distribution is a distribution because it has a package manager, a shell, and its own OS-level utilities or different kinds of dependencies. That's what keeps a distribution a distribution. Let's say if you are specifically writing an application where we have seen in the past the multi-stage builds, where all the tools used to build a container are removed once the build is done because you don't need them in the runtime. This ensures your container's startup time is good and efficient. The same thing goes with Distroless where your focus is that whatever is needed for the application, whatever libraries are needed exactly with the application, you just need them and you don't want any other OS-level dependencies, package managers, or shells. This essentially strips down a lot of things and ensures the container size is very minimal, maybe 20-30 MB depending on what you add. So, in brief, this is a Distroless image.

Bart: And what's not to love about smaller container images?

Harsha: Of course. I think if we look from a developer perspective, all they need is efficient containers which can handle spikes in traffic. They want their pods to be scaled in no time. For example, from 50 to 500 in the next two minutes. If you have a heavy container running, even if it is cached, it still takes time to load the application and bring it up. If the container is as minimal as possible, the start time, the space it takes in your registries like ECR, and the pull time are significantly reduced. So from a developer angle, that's a statement we should agree on.

Bart: If someone is a security-conscious individual or company, should they consider using Distroless as it has fewer binaries? It probably means less chance of getting hacked.

Harsha: Yes. The simple fact we have to remember is fewer binaries and libraries equal fewer vulnerabilities. Newer libraries add functionality but also introduce more vulnerabilities. So, the fewer binaries and libraries you have, the fewer vulnerabilities. Security should definitely consider using it, but there's a myth that Distroless containers are secure. This is not true because certain libraries still lurk around. Just removing a shell does not mean you're safe. Good point, but let's ask it in a different way.

Bart: If someone's image doesn't have a shell, they can't attach it to the running container and debug it.

Harsha: If we look at shell in a container or in general, shell is, I think, after GUI, the next easiest way to interact with the system. Shell is like a couple of abstraction levels. Essentially, it is a couple of binaries making certain system calls. The same system calls are also made by your application, your application libraries, and functions. The same kind of system calls are made. Removing shell might make it harder for an attacker to do what it does with a shell. This does not mean that he can't do anything without a shell because if the same level system calls are made with a different library, it's the same thing he's able to do. End of the day, his motive is getting in. Removing a shell itself does not mean that you are safe from executing all the commands. With that in mind, are Distroless images convenient primarily for size and less for security? I think it's both. Size definitely, and security also because if you look at a normal distribution container like Alpine or Debian, it has a lot of OS-level dependencies, package managers, and stuff. It is just that it has more libraries versus a Distroless distribution, which has fewer libraries. You are cutting down on the attack surface, definitely, but you can't call it a secure container and relax. You can't take it for granted that using Distroless means you don't need to worry. That's not a true statement.

Bart: Most of the benefits you mentioned can also be found in Alpine-based images, which are much more secure thanks to musl. So why even bother with Distroless?

Harsha: I think the thing with Alpine again is there was that hype of Alpine where I think two years back a lot of people were using Alpine as a hype prompt. I think a simple thing Alpine did was we have GNU libc where Alpine came up with musl libc, if I'm pronouncing it right. It is basically a stripped-down version of GNU libc, and certain implementations were rewritten. It is essentially doing the same; I mean, the direction is also distribution-less, but it is retaining its distribution aspects. It has a package manager, a shell, OS-level dependencies, and all these things, but it ensures that the container size stays less. But over time, I think we have seen a lot of issues with Alpine as well. There was an article on how there were DNS resolution issues in Alpine, and I think there's also an article on how memory allocation is done differently in Alpine. They use the best fit allocation memory model where it tries to allocate the largest chunk possible at a single go, and it leaves a lot of gaps in the heap. So there's an article on that. So, if you are trying to bring down the container size by making the system kind of a bit unstable, I don't think it's the right way to do it. Instead, maybe you can look at a way where the container should be as stable as using a full-fledged OS. At the same time, it should have less attack surface. Now, Chainguard came up with things like Wolfi or these kinds of images. Maybe something like that.

Bart: Can we take this to the next level? Let's see. If someone really cares about security, shouldn't they use empty containers? And why does someone need a base operating system like Alpine or Debian in the first place? Empty means no binaries, no problem. Isn't that right?

Harsha: I think if a developer exactly knows what they are doing, using an empty scratch container and pulling in all the necessary libraries for the system to be stable is okay. But if you do not know what you're doing, using empty scratch containers might be a problem. For example, if you're missing /etc/hosts or there's no... /etc/passwd file or there's no time zone defaults, you will fail on logging and a lot of other things. Even cron jobs and many other processes depend on the time zone info. If these things are missing in the cache, you might have a hard time. That's why they try to use certain distributions where basic stuff is included, and at the same time, they can run their applications.

Bart: We started off the podcast speaking about Distroless and have now mentioned Alpine and Scratch. Are there any other base container images that follow the same philosophy and can help folks keep their size down while still helping them maintain their security posture?

Harsha: I think as I mentioned, I've come across Chainguard's Wolfi image and there's one more from Ubuntu. I've not used it. I think it's called Chisel or something. These images ensure that they are production standard bits at the same time. I think Chainguard also claims that it's a zero-CVEs image. But I don't think that's the right metric for seeing that a container is completely secure. Still, compared to other distributions' containers, I think Wolfi is doing a great job. So maybe you can consider using Wolfi.

Bart: In terms of takeaways from this conversation for our audience, security doesn't seem to be something that you can turn on or off, but rather a continuous practice in which you have to invest a significant amount of time and effort. You showed us how to exploit Distroless container images, which are supposed to be secure. What would be the advice to our listeners who might not use Distroless but still want to secure their images?

Harsha: Snyk, I think securing containers is not just about securing containers. I always bring up this, where I see infrastructure in seven layers, right? There is a CDN layer, there's a cloud layer, there are clusters, there is compute, you have both server and serverless, and there are containers and there is code. Underlying all these layers, there is compliance, where each layer should be compliant with various frameworks and benchmarks. So when you mean securing a container, it does not just mean that you are securing a container because it is always dependent on a layer above or a layer below, depending on where you're deploying it, how you're deploying it, where your source is coming from, be it a library or anything. It involves regular container security best practices all the way from choosing a base image. How are you managing your dependencies? Are you having SCAs? Are you checking open source packages? Are you creating SBOMs to check? Are you creating that breakdown of the whole application and understanding the different vulnerabilities of all these activities and what checks you have at CI and CD level? Are you signing your layers? Are you trying to validate the supply chain aspect? Are you trying to validate where these direct libraries or even transient dependencies are coming from? Who's doing the code review? These branch protection rules, all the CI and CD aspects of it, and maybe coming to the runtime while deploying. Do you have admission controllers deployed? Are you checking if the right kind of base image is being used, if the whitelisted base images are being used or not? Are you trying to pull an image out of the defined scope? Maybe you told that, hey, just pull from this ECR and you're trying to pull from Docker Hub. Is that allowed? And coming to runtime. Maybe network policies of pods, what can talk to what, and what user you're running the container with. The network calls a container can make, are you letting people exec into the container or letting them debug using a kubectl debug or creating a temporary pod? And coming to the runtime against the code. How are you monitoring these containers? You have followed all the best practices, but there's an application running on the container and somebody exploits it through layer seven. Somebody does an SQLi or a command injection or XSS. Are you monitoring it? Is there a WAF? If there's no WAF, what are you doing about it? These are all the best practices you should be following, not just specific to containers but all the layers I have specified, which are interdependent with the container layers.

Bart: Most of the tools and practices we discussed were around base images and protecting against runtime vulnerabilities.

Harsha: I think most of these things are the indirect attack surface area. The way we used to see this attack surface area was always solely focused on the perimeter or internet-facing assets. Right now, because there's a lot of sophistication already in the internet-facing layer, you have apps, very smart labs, and active apps where they tune their roles on the go using machine learning algorithms. The attacker is also having a hard time, so he's trying to find certain ways to inject things into the system through unusual methods. So we arrive at the supply chain. He tries to identify what are the transient dependencies. We've seen the recent attack as well. If he could pull that off, he could get into the system without trying any reverse shells. He could just get into the system and persist his access through a single library or an injection from the supply chain world. What's important here is signing your artifacts. You can use Cosign or something similar. It's not just libraries; it should be signing layers, signing your image layers. When there's a multi-stage build, we end up scanning the final build, but what if there's a transient dependency in any of the layers before the final build? How are we covering that? Again, coming back to the runtime part, we should have Kyverno. If we look at that, there are great policies out there, both written by the community and pre-existing, even in the supply chain area, where it checks what is a signed image. Where are the sources? Is it a known source or not? What kind of base images are you using? Are you trying to bypass something? You can even write a lot of custom policies using Kyverno. In the supply chain aspect, we can mostly look at these aspects.

Bart: Now, you write a lot, apart from the blog that we've spoken about in today's episode.

Harsha: It does. Like I said at the start, I think 99% of the things we learn are from the community. Whether it is YouTube, Stanford CS50, or Stack Overflow, it's mostly community. If you think you know something really well, that's the best way to give back to the community. It's also a very good reminder and validation of what you've learned. In the last few years you've been in the industry. The reminders I get are like, "Hey, these are all the things you've learned in the past." That makes me... Recently, I've been talking on these kinds of podcasts.

Bart: We found an old LinkedIn post of yours where you made the case that sharing your knowledge is better than gaining certifications. Do you still believe that?

Harsha: I always believe that because I'm not a big fan of certifications. One thing is, the ratio of certs to courses is you buy one course, you get 10 courses. So you get one cert at the same price, you get 10 or 20 courses at the same price, right? It's not just that. But courses, see, basically the system of certifications is that. I have to prove that I know something, and we know that certification systems are broken. Right now, if you see that I'm a CH, an OSCP, or whatever, you can't vouch that I have certain skills just by looking at my certs. Instead, if I can prove it by writing, talking about it, or helping others, I think that's a great way to show what you know. At the same time, it helps the community as well. In fact, in one of the old interviews I was part of, the interviewer had read my article on why SSH proxy forwarding is a bad idea on Medium. He recognized that it was me. That has a different kind of impact, I feel, versus a guy who holds 10 certs and puts it on LinkedIn or whatever. I think there are also mandatory rules from companies that you'll have to do this cert. It's okay, you do it as long as the company is paying for it. But if you're spending thousands of dollars on certs, I'm not a big fan of that.

Bart: In addition to writing, you also have your own podcast.

Harsha: I've recently started a podcast called Kernel Space. I wanted to start it about two years ago. I never pulled it off, but I just started it. It's mostly conversations with CISOs, security founders, or security researchers. I try to network with them and understand certain aspects which are less spoken about. In simple terms, that is what the podcast is all about. So, yeah. Good.

Bart: I also found out you're going to be speaking at BlackHat USA 2024.

Harsha: So we have built a tool, my colleague Anand and I, called Cobra. It's a cloud offensive breach and risk assessment tool. It simulates multi-staged attack paths. For example, we've seen many attacks where an attacker gets into a system and escalates privileges, accesses data, and maintains persistence using the matrix framework. It focuses on the cloud-native environment. For example, a web application gets exploited, the attacker accesses secrets, takes over the cluster, gets into the node, and takes over the role, leading to account takeover. These are all chain attacks that are happening. We built a simulation tool around this so you can evaluate CNAPs. For example, if you're doing a CNAPs POC, you can deploy three or four attacks from GCP, AWS, Azure, and K3s to see if they can detect them and what controls you have in place. This is the tool. It got selected for BlackHat Arsenal and Defcon Demo Day in Cloud Village. Excited to be there and present this.

Bart: What's next for you? Apart from podcasting and speaking at BlackHat, anything else?

Harsha: I'm focusing on certain aspects of research. I mentioned one use case where you steal an etcd pod, access secrets from it, and try to put something in etcd to bypass the admission controller. I'm experimenting with a few things. Apart from that, it's general security stuff. I'm also trying to build a security community on Discord. We have about 50 people now, with a couple of very active members. These are the things I'm doing these days. If people want to get in touch with you, what's the best way to do it? LinkedIn. I'm active on LinkedIn and I also use Topmate. Topmate is a one-on-one mentor-mentee platform where I do it for free. It's basically to expand my network and talk to great people. If I don't know something, I learn from them. If they need help, I go there. So, yeah, I keep doing this Topmate stuff. I'll get old stock.

Bart: Fantastic. Harsha, thank you so much for your generosity regarding your time and how you're sharing your knowledge. I'm sure you'll be inspiring others to do the same thing. We'll be looking out for the next things that you'll be sharing about security. In the meantime, take care. Thank you.

Harsha: Thank you so much. Glad to be here.

Bart: Cheers.