Supply chain security: bridging the gap between developers and Kubernetes

Bart: Who are you, what's your role, and who do you work for?

Tim: I'm Tim Miller, CEO and co-founder of Kusari. I've worked for Kusari.

Bart: What are three emerging Kubernetes tools that you're keeping an eye on?

Tim: There are a couple of tools that we like around Kubernetes. The tools that help developers get into the ecosystem better are the ones we really like. Ko is one of the tools we really like, as it helps developers get their code up and running in Kubernetes with ease. We also love many of the Sysdig tools, such as Falco, which are really cool for understanding exactly what you're dealing with. Our focus is on tools that provide visibility, and we find those really great. Additionally, we've been finding tools around SBOM generation that are making things more transparent in Kubernetes. Some exciting tools are coming out of Google, such as Excalibur. They're also partial to tools like Guac. Overall, DevTools are great.

Bart: Now, supply chain security, particularly the topic of SBOM. One of our guests, Harsha, explained that container security doesn't stop at using minimal container images. Instead, one should look at the entire supply chain, dependencies, software composition analysis, and software build materials. What's your experience in securing the supply chain?

Tim: Those two ideas are complementary to each other. Minimal containers are a fantastic thing to do, and everybody should be doing it. However you're able to achieve that is great. Minimizing things has been a pattern for a long time. You don't want extra shells in your containers or things that can increase the surface area, leading to more problems, not to mention issues with efficiency. But minimizing containers is not the only problem you have, unless all you do is take container images and run them. Past that, you still need to understand what your software is, where it came from, and how it got there. That's really where a lot of problems pop up, particularly in the provenance of things. The Software Development Life Cycle (SDLC) is designed to simplify that information and make software development easier. Deployments, packaging, and removing complexity are part of what many toolings are trying to do. However, that's the information you need to fix something when a problem arises. So, keeping things minimal but also knowing and keeping track of things is crucial to being effective, especially if you're dealing with more than one project at a time, which I think basically everybody is.

Bart: Container images and vulnerabilities. Our guest, Tim Miller, debunked a myth that minimal container images are more secure because they have fewer binaries. He explained that the system calls made by the shell could be issued by another process and lead to an equal result. He suggested that minimal container images aren't enough. What's your view on this topic?

Tim: I think it's pretty similar to the previous question. I agree that minimal container images aren't enough. Part of what you're looking at is what those binaries in that container are actually doing and whether they're necessary to perform the function. In general, something like a shell is almost never going to be in a minimal container image. You almost never need that. You want to get rid of those unnecessary things that can elevate permissions and do extra stuff that you really don't need as part of the job of that container. I totally agree that those should not be in there. However, I think it really goes back to making sure the things that are going into that container are exactly what you intend to complement the job that you're intending to do. Separately, I'd ask, how do you actually know that the container images are in fact minimal? This goes back to the transparency and understanding question. You really have to keep your eye on both transparency, the collection of things, and the minimalization of things. It's all part and parcel of doing this well.

Bart: Secret scanning security. Yakir and Asaf, two of our guests, emphasized the importance of using multiple secret scanning tools to identify different vulnerabilities. They highlighted that each tool uses distinct methods, some of which are regex-based, while others are entropy-based, yielding varied results. What's your advice when it comes to securing and scanning for threats?

Tim: It's pretty hard, and you should probably use all those things. Generally, though, what we found is that the tooling around finding secrets in your repositories is less important than making sure you have the right processes in place to prevent them from getting there in the first place. Things like SLSA attestations, verifying that you've got the right permissions, and ensuring that developers don't have access to put in secrets are crucial. If they do leak a secret, it should be short-lived and single-use, rather than a generic open-door password. I'm sure a few people have PTSD from that. The general way to approach this problem is to focus on the process that allowed secrets to get there in the first place. While tooling for finding secrets is important, it's secondary to addressing the underlying process issues.

Bart: Someone I was speaking to earlier was talking a lot about automation. He said that security falls short when it comes to automation. That might sound a little general. What's your take on that?

Tim: I think the way folks deal with security automation is kind of funny. Security is often seen as stopping you from doing something. When people say security falls short of automation, I think they generally mean it's not enabling them to move faster. It's like putting up gates to prevent bad things from happening, such as secrets ending up in a repository. Many tools in the security space are designed to put up walls. However, your ability to react is probably your biggest security tool. Enabling folks to move quickly is a significantly undervalued security perspective. I would generally agree with that.

Bart: When it comes to Kubernetes, what's your least favorite security feature?

Tim: Generally, being open by default is not ideal. The amount of effort required to make a Kubernetes cluster usable in a real environment is the biggest problem. On the flip side, this openness is also what makes it easy for many people to get started, as everything works out of the box. However, this is not a good model to start with. Fortunately, many tools in the ecosystem can help address this issue, and there is momentum behind paid solutions to fix it. A more granular access control would be well appreciated by everyone.

Bart: For any developer working with Kubernetes, what's the one piece of advice they simply cannot do without?

Tim: Infrastructure is really hard. I would look for tools, as there's a lot of tooling in the space, and a wealth of information to get started. I would not jump right into being a Kubernetes administrator. Instead, if you're a developer, start by looking for ways to get your application into the Kubernetes cluster, and then move yourself up the stack from there. Many folks fall into trying to understand every little bit about Kubernetes under the hood, such as how to deal with access controls and scaling policies. I would focus on getting the basics around deployments going first, making sure you understand the models. Keep it small, as Kubernetes is designed to take something, scale it, and make it easier to manage. Don't put in huge things; make those steps pretty small. Start on the dev side, not the admin side. I think that's been the biggest overwhelming fact we see with new, frazzled Kubernetes users.

Bart: Kubernetes turned 10 years old this year. What should we expect in the next 10 years to come?

Tim: I think a lot of these themes will hopefully be true. I hope to see more automation around enterprise-friendly Kubernetes, particularly without the high cost. This has been a barrier to entry. For example, you see things like OpenShift, which is unaffordable for most people. Smaller companies need something more approachable than true vanilla Kubernetes, but without all the bells and whistles that come with a product like OpenShift. I'm certain we'll see more of this. Additionally, I expect to see more development in the dev cycle and how to get into it. I expect tools like Flux to take off, helping to simplify the CI/CD aspect and making it easier to get into Kubernetes.

Bart: Particularly on Flux, and also a minor shout out to Andy, thinking about GitOps, security - where do these two things come together? Can you tell me about the security aspects of Flux?

Tim: I think the general model is how I would approach this. The fewer opportunities you have as a human to affect the process and how you get something from A to B, the more automation you have to do to validate that and make sure everything is working ahead of time. If you invest in your automation and workflow, it's going to be far more repeatable. This also makes it easier for people to come on board. I think that's why folks adopt this model - you don't want to have to teach every single person who comes onto your company every single little thing that you have to do to get your thing from A to B. Tools like Flux and the general GitOps model are beautiful in this way. They do require more upfront investment in the workflow, but I think that's a very worthwhile investment for basically everybody.

Bart: What's next for you and Kusari?

Tim: So next for us and Kusari is really getting our SaaS platform off the ground. We just started our go-to-market a few months ago. We've been working on an open-source Guac for a couple of years now, and after listening to what everybody's been saying for so long about what they love and hate about Guac, we've put that into a product. Now, I'm focused on getting that out there, listening to what people have to say about it, and iterating until we get it right.

Bart: How can people get in touch with you?

Tim: A bunch of ways - Kusari is a great way to find us. We're on LinkedIn, there's the Open SSF Slack as well, you can find us at Kusari. Generally, Kusari, you'll find us there. Picture of my daughter on my shoulders.