Bart Farrell: First things first, who are you? What's your role and where do you work?
Andrew Martin: Hi, I'm Andy Martin, founder and CEO of ControlPlane and cloud-native security aficionado.
Bart Farrell: Andy, in a regulated Kubernetes environment, where should the hard security boundary actually sit? The cluster, namespace, workload identity, or cloud account level? And what failure modes do you see when teams rely too heavily on Kubernetes RBAC as the primary isolation mechanism?
Andrew Martin: It is difficult to enforce hard multi-tenancy in Kubernetes. It was inherently designed as a platform and therefore things like nodes are not namespaced. We can have node pools into namespaces, but the namespace concept inside Kubernetes doesn't extend uniformly. RBAC role binding is also complicated. It is easy to make mistakes there. We actually built some formal models to prove RBAC configuration because static analysis just does not work. The difficulty there is we want to shift left. We want to isolate the process with the lowest-grade permission set that's possible, the same way as we assume breach and assume insider threats. We can start with Kubernetes clusters, but a very large multi-tenant cluster is a risk because shared data classifications are okay. Mixing data classifications, you break out of a non-sensitive environment, you hit the kernel and then you can probably pivot onto another node. Cluster level full hard multi-tenancy is very difficult to achieve. Namespace is useful when we come to access controls. Obviously, our network policies are there, but our horizontal pod autoscalers are also there. Identifying what is a namespace from a Kubernetes namespace concept for an application design reason versus a security reason is also a healthy friction that is constant. Workload identity, one of my favorite things. We have the same problem of trust domain segregation. If the application or the organization is not modeled in such a way to identify or cross between trust zones when required, you can also bleed out identity in that case. The cloud account really becomes the avocado that we can scoop out the entire infrastructure and redeploy it again. If we can treat our cloud environments as independent individual avocados, then we can install the entire infrastructure in them, we can blue-green into the next one, and have end-to-end repeatable and testable infrastructure as code. Of the multiple layers, the most important is to have that full stand-up and tear-down in the cloud accounts, and then we can build the other layers on top of it. But none of them independently will do the full job.
Bart Farrell: Andy, how would you design break glass access for a production cluster so that emergency privileges are ephemeral, strongly authenticated, cryptographically attributable, and fully reconstructable after the incident without leaving standing admin credentials behind?
Andrew Martin: The reference behavior is to repave any infrastructure component that's been touched. Ideally, we have an immutable base image. We have kernel modules that can immediately dump all the memory for forensics purposes. And the access comes through a temporary step-up through a centralized IdP. The difficult portion there is then to have a fully reconstructable incident trail. We are relying upon the intrusion detection systems that we've already got deployed, and probably those that notified us that something was amiss in the first place. In the event of compromise, we cannot always trust the IDS post-compromise. Who knows what actually happened? This is where immutability forms a useful base layer. The full reconstruction is a property of the tampering that's occurred post-breach. For the rest of the pieces, take the machine, dump all its memory, image all its disks, shoot the thing in the head as soon as you're done, and ensure again that we're doing cattle, not pets, and that we can scoop out whatever size fruits we have and redeploy the entire application or the entire node or the entire cluster, or the entire cloud account for disaster recovery, post-breach, feature delivery, scale-up, region failure. These fundamental everything-as-code properties from infrastructure, policy, and security mean that everything is OCI-defined, everything can be redeployed easily, and it gives the security properties and guarantees of the system then bleed through into the performance and stability layers as well.
Bart Farrell: For a bank that's trying to establish end-to-end workload provenance, how would you chain together source identity, CI attestations, image signing, admission policy, workload identity, and runtime verification? And which parts of that chain are most commonly treated as trustworthy when in fact maybe they shouldn't be?
Andrew Martin: This is something we've been sponsoring PhDs for and deploying solutions for many years. Fundamentally, the composition of the source that runs in production is first-party code that I've either hand-cranked or had a machine do for me, composed with some third-party code. The first stage to bring that code into the organization and ensure that its provenance, veracity, its own set of transitive dependencies, and any further artifacts that it brings with it, including upstream attestations, are packaged and available for the bank. Once we move through into the developer workstation and their developer process, the SDLC moves into the CI. This is the most trusted high-risk component in any organization generally. The entire SLSA supply chain artifact security levels was built based on how Google built their data centers. Every data center is FedRAMP-approved. They have exceptional step-up and control, and the entire infrastructure is built to be a uniformly addressable cell, built on the Borg concept, so that they're self-contained, but they're part of a massive distributed system. The security properties of those systems are not transferable into, I hesitate to say commodity, non-hyperscaler data centers. That means that CI system generally has backdoor access. There's probably no intrusion detection on there. Once we're moving at machine speed, which means as fast as AI can infer, for vulnerability analysis and attack tool calls within other people's systems, we need to be as responsive as we would be for production infrastructure in our CI. The next part of that, when we're looking at the image signing attestation, all of these things can be built through the in-toto protocol and on top of that we sponsor a project called Witness and that comes with something called Archivista. At the point that we start rubber stamping everything that comes along the conveyor belt saying at the time we trusted what was in this because we had a look inside it, that rubber stamp is a point in time affirmation of enjoyability. We enjoyed working with it, threw up no red flags. A zero-day may turn up today or tomorrow, which means that artifact is no longer suitable. However, we do know at least that the composition of the things inside it is as described. It doesn't stop vulnerability, but it does guarantee that the recipe is signed by the chef. Using those attestations, which are all cryptographically signed from a central root of trust managed by the security team, we can build up a chain of provenance of the software that's going into production. And at that point, everything else becomes easy because we have immutability throughout. We have cryptographic guarantees based off that immutability. And we have the trust from the humans who've been involved to identify whether or not those stages were what they liked at the point in time that they're coming through. At that point, admission policy is the validation of a single meta signature to say the whole pipeline is validated. The workload identity is a function of the application with whatever labels it has running in production, so we can safely give that workload identity to this composition of inert code and allow the two to run in a way that includes credentials and therefore network access. And for runtime verification, because Kubernetes inherently never allowed the mutation of a pod, we used to be able to rely on admission control, because if we scale out, we'll go back in through the front door and we can say admission control will hit all pods. With the advancement of in-pod resizing, we may now have pods running in production for weeks, months, years. We've had to then implement revalidation and compliance operators to make sure that the composition of the attestations in relation to the code is still checked with a recency. That means, if a zero-day comes out. We need to know that not only that it's there, but the configuration doesn't support it, or perhaps there's a different configuration now needed to pass that off as secure. It all starts with CI, and then those attestations can be used to guarantee other security properties of the system, as long as we're operating within a cycle and a heartbeat that supports the risk tolerance of the organization.
Bart Farrell: Andy, what's next for you?
Andrew Martin: The most interesting things right now are automatic programming, how the world has inverted itself from carpal tunnel keyboards to throat nodules from singing too much into AIs, having the transcription do the work. The nature of collusion and breakouts for these foundational models, we've essentially found ourselves in a situation where we have nation-state actors in a box behind the firewall, and as we use models with no provenance, even Western models have no public provenance. The risk level there is firstly unmanaged and secondly at an exponential scale that's far more than we've ever experienced as an industry or really as a species. We've been working on detecting collusion. The breakout prevention for these things is gold-standard cluster configuration which we're very lucky that some of our clients will agree with us and say the risk of this IP, so trading algorithms for example, leaking out means that we will secure this to whatever level we're recommended to get the highest degree of confidence and trust in the system. For many organizations, the balance between feature delivery and securing a system is diametrically opposed. If they can't ship features, their competitors will take the market share. If they are insecure, they might get away with a slap on the wrist from the SEC. We don't really see much punitive action against the large examples of social security number leakage, Equifax and things over the preceding years. As it stands now, everybody is suddenly very interested in what is the gold standard for cluster security configuration. Working in that direction, I spoke about the formal methods for Kubernetes RBAC, but that is part of the agentic protection systems that we're building, the governance and credentialing. ControlPlane sponsors Flux CD, which is perfect for running huge swarms of agents. It also sponsors the OpenBao product or project, which helps to fix the identity problem. Right now, non-human identities are proliferating at an exorbitant rate. We have foundational technology, workload identity, again, in SPIFFE/SPIRE, but we know that SPIRE is difficult to deploy. We've built out extensions for OpenBao that just take all the pain out of running OpenBao as a central workload identity provider that can also bridge through into human identity. We're really looking at how do you trust an AI on the basis that we don't trust it in the slightest, which is an interesting philosophical experiment.
Bart Farrell: If there's one question that you would like to have answered when it comes to running AI workloads Kubernetes in highly regulated environments, what would that question be?
Andrew Martin: If nothing else, I'm certain that things will not get much easier in the next six months. The asymmetry between attackers and defenders has moved at exponential scale, and it will be a tricky six months. Nevertheless, the interesting thing that I suspect will happen at the end of the year is with two major IPOs, those IPOs must hit back because the Western financial system is overly leveraged on them. Chinese models will race to open source, competitors of Kimi K3 or Qwen-plus. That would be a very interesting end to the year with the advent of potentially local models. The M5 Max is shipping 128 gigs of unified memory in a laptop. There is huge opportunity for local inference and also local end-user device protection that will emerge over the next year or so, which will be very interesting indeed.
Bart Farrell: If people want to get in touch with you, what's the best way to do that?
Andrew Martin: Come and see us at controlplane.io or at any of the community groups, working groups or CNCF conferences or any of the summer security camps.
