Bart Farrell: Welcome to KubeFM. Who are you? What's your role? And where do you work?
Manfred Moser: Hi, my name is Manfred Moser and I'm a DevRel Engineer at Chainguard here. We're remote all around the world.
Bart Farrell: So, Manfred, in a regulated Kubernetes environment, what does a genuinely verifiable software supply chain need to prove at build, artifact, and deployment time? And where do signatures, SBOMs, provenance, and admission control fit?
Manfred Moser: Ideally, you'll have full proof for the origin of all your components and that's a lot of documents that you need to understand and look at because typically you have a lot of components involved in your whole supply chain. The problem with that is that in that supply chain you need to be able to trust all the actors in that chain and that's typically many. And the supply chain attacks we've seen in the last couple of years, especially the last two years on the NPM registry, for example, show you that a lot of actors are not trustworthy, there's malware deployed in the NPM registry, constant attacks happen where maintainer credentials get stolen and then ghostware gets installed that has credential theft into your supply chain. You can't really trust the whole supply chain. The best way you can deal with that is really to try to reduce the supply chain to actors you can trust. Chainguard provides that because we have one vendor that supplies the components all the way. OS packages, container images, but also libraries. We rebuild them all from source and we verify them. So you can sort of cut that complexity of who do I trust down to just us and therefore have a much simpler picture of what to think about.
Bart Farrell: When responding to CVEs or a zero-day at scale, what are the hardest technical problems? Rebuild latency, transitive dependencies, patching strategy, compatibility, or something else?
Manfred Moser: This might sound trivial and silly, but all of those are very important. What it boils down to is that the biggest problem is time. You don't have time to think about what's the best strategy, how we are rebuilding this nicely. These zero days and incidents come faster and faster sometimes even before public vulnerabilities are disclosed, so there's always time pressure and you can therefore as a result whatever you do and what you have done also in the past on those incidents is you probably have not done a full proper job. You kind of as you said, rebuild latency, you're patching around and hoping for the best. What really helps you the most is if you give time outside of these incidents to make your infrastructure and your applications much more modern and much more secure. So you need to do the maintenance, need to upgrade to the latest available components. You need to proactively reduce your CVE footprint to near zero. Don't just look at the critical severities, get all of them eliminated, go to the latest versions of everything. That's, of course, always outside the priority that people talk about when they want to upgrade your applications. But ultimately, that's what you have to do. We provide a lot of that work. Our Chainguard containers, for example, always use the latest components of everything. And what that means for us is we basically rebuild our containers nearly daily because some random component always comes up with a new release. And that means often when it's widely used, we're rebuilding thousands of containers a day. So the work has to be done. Ideally, you don't do it all yourself because as you say in our blog post, so from this shit is hard. So it's a lot of work.
Bart Farrell: Well said. And where should policy actually be enforced in a bank-grade Kubernetes environment? CI, registry, admission, runtime? And also, how do you prevent policy drift or conflicting controls across those layers?
Manfred Moser: Unfortunately, it's a complex story and the answer is not simple. It's not a choice. You can't make sure that our components are fine when we ingest them. That's not good enough. You have to enforce your compliance and your component security throughout the system at all places and also throughout time. For example, throughout time, that means just because a component that you ingested three weeks ago and was the latest and greatest and had no CVEs and was great then, that doesn't mean it's good today. New CVEs are found, unfortunately, even more rapidly now with the AI tooling than they were ever found before. Change is increasingly rapid. You have to verify and enforce your policies all the time. The second part of that is that it's not good enough to enforce them at the ingestion, or in CI/CD or in production. You have to enforce them everywhere because these attacks and malware exploit developer workstations. With people vibe coding and using AI tools all over, they exploit random machines. They target CI/CD because a lot of secrets are stored. They are there to access production and then they run all over the place. Unfortunately, the answer is not simple. You have to do it throughout the system and all the time, which goes back to what I mentioned earlier. Ideally, you'll keep all your components up to date as much as possible and also as simply as possible. So don't use random big containers with stuff on it that you don't need. Reduce your applications to use the libraries you need only, make them modern and so on.
Bart Farrell: If people want to get in touch with you, what's the best way to do that?
Manfred Moser: Our website is chainguard.dev and you can find me on LinkedIn and everywhere else on the internet.
