Bart Farrell: First things first, for folks who don't know you, who are you? What's your role? And where do you work?
John McBride: Hey there, my name is John McBride. I am the CTO and co-founder of the Paper Compute Company, where we're building some AI agent gateways and observability systems. I live in New Haven, Connecticut on the east coast of the United States, and been a Kubernetes nut for many years since the Tanzu days at VMware, actually.
Bart Farrell: Very good. Well, good to have you with us. As AI agents start diagnosing and potentially fixing Kubernetes incidents, which operational actions do you think we can safely hand over to an AI agent today? And which should still require explicit human approval?
John McBride: Kubernetes is already very good at this with RBAC role-based access control systems that are already baked in and were purpose-built from the days of containers of different privilege levels, being able to do different things on the clusters that you had. So I sort of think about it in three different areas. Obviously, agents can be far more autonomous than any kind of software systems we've ever encountered in the past. The really good area that I think we could start giving to agents today are things like read requests and just general diagnosis. Like I need to go read a bunch of logs, read a bunch of traces, read a bunch of stuff across things on the cluster to just figure out what's happening. And you can give some just broad general read access to those agent operators on the cluster. There's also some interesting things that I've seen with some scale-up actions like non-destructive scale-ups. If you are seeing like a big spike in something happening and maybe you don't have certain things configured correctly to scale up for that kind of traffic controls, you can actually let an agent start to manage those controls itself. I've heard of some rather large companies doing a lot of big Kubernetes things, thinking about this from an interesting agentic perspective, because there's all different kinds of things going on, you know, be it agent traffic hitting your services or your clusters, that maybe you would need to scale up for that don't necessarily look like typical traffic patterns. So you can approach it from an agentic point of view as well. I think another thing that is maybe soon that we could start to give agents as people continue to build out these systems, are more deterministic, destructive actions. So this would be things like, oh, a pod is failing or is in like a CrashLoopBackOff because, you know, like an image isn't available that hasn't been released or hasn't like made it through CI/CD pipelines to be available in your image registry. Oh, maybe the agent could take action on that to roll back a release or actually try to heal the cluster a little bit. Karpathy last year or a year or two ago said something about like self-healing infrastructure and self-healing software. And the first thing I thought it was basically this: where it's like, oh, an agent could probably just fix a CrashLoopBackOff pretty quickly if there was, you know, a pretty obvious configuration error or something. But I would still put that in the category of like somewhat destructive actions because you're changing the sort of state of the whole cluster. So I think about it with the lens of, you know, wrapping a lot of that in bigger, I guess, validation loops or maybe what people would call like software factories today. And I think that the Kubernetes lifecycle, the Kubernetes software development lifecycle actually fits really nicely into this whole idea of software factories, which, if you're not totally AI-pilled today, all that is just a fancy rebranding of big CI/CD with maybe a touch of end-to-end tests in there. So if you already have some clusters somewhere, if you already have like staging clusters, or even kind clusters can be a use case for this. If an agent can just spin up the kind cluster, apply some sort of like local manifest to it that looks close enough to production and then it can poke around and prod around and make sure that it's going to do the right thing. That I think can be a good sort of start to building a bigger deterministic loop inside of a software factory to do some more of these somewhat destructive actions. Some of the things I would not give to an agent, I think this is also worth calling out, is things maybe that fall under like a cost center. So like if you're going to like bring up a bunch of clusters or start spending, you know, if the agent's going to start spending many hundreds or if not thousands of dollars on new actual infrastructure or nodes or compute. I would not give that to agents today. I think the human needs to stay absolutely in the loop on that because they can run away with bills and they will run away with bills. So be warned. The other thing I would say is anything break glass that maybe falls outside of good old-fashioned, you know, GitOps or things that you wouldn't maybe put in a manifest. So manual destructive actions in a break glass situation. And then there's some things that I would say never, ever give to agents. And that's stuff. And maybe this is a hot take, but that's stuff that like we're already good at. Like I wouldn't give node scaling to agents because Karpenter is more or less and, you know, Karpenter-esque projects, I would say more or less have like solved this. And I don't see like the need to throw in the non-deterministic agent loop inside of there, even though you could probably build it and spend lots of tokens doing it. I would just argue like, well, there's good projects out there that exist that solve these problems with software, instead of large language models. Another good one maybe is like multi-cluster control planes for something like Crossplane. That's a very well solved problem, I would say. Obviously, a lot we can do on top of that, or even more agentic things on top of those things. But I would not give agents the levers to do things that exist as good projects already.
Bart Farrell: Traditional Kubernetes observability tells us what happened in the cluster. But if an AI agent is making operational decisions, what should teams capture so they can reconstruct and audit an agent-driven incident response after the fact?
John McBride: This is a great question. It's so critical because you can really quickly find yourself in this position of what the heck did the agent do? Even as you start to give agents more levers, you know, even in just read-only contexts, you start seeing all these like read requests going through or like traffic spike up because of your agents. The most benign ways you're going to want to know, like what are these agents doing? There's another angle to the whole software factory that people talk about, which is the dark software factory. And that whole idea is that like, well, you know, the iPhone can be made in a dark software factory where they literally turn off the lights because it just goes through these assembly lines and on the other end you get an iPhone. They've automated it so much that you don't even need people in there to see with their eyes because the lights are off. That analogy being that like you could make a dark software factory where then you wouldn't even need to look within it and you just kind of go totally hands off. I disagree with that and I think that you should be able to have the mechanisms to know and understand like what your agents are exactly doing when you need to know what they're doing because in an incident, if something goes off the rails and the agent causes something to maybe get worse, or you're just unsure of what it was doing, but you're going to want to know. So there's a lot of stuff that you can do to make this possible. Most gateways today support some kind of session logging, if not just grabbing all that trace data. OpenTelemetry, I think has become like one of the biggest and best adopters of this with their GenAI semantics. Today you could basically start grabbing trace data from your agents in OpenTelemetry and put it wherever there's an OpenTelemetry Collector-compatible store that you probably already have. There's some interesting stuff happening with this other kind of semantic called OpenInference, which is also really interesting, worth looking at. But I think a lot of people are waiting for, someday something will arrive that we can start capturing this data with, right? And it's arrived. It's here. OpenTelemetry, I think, is going to solve this. Even better yet, if you're doing inference on your own cluster, i.e. if you own the end-to-end request through some gateway or some AI gateway to do a model selection into inference on your infrastructure, maybe you have GPU capacity and you're running something like VLLM. And then, you know, it does the inference, it returns the callback, you know, it's not going out to some third-party providers like OpenAI or Anthropic. But if you have that inference in-house, if you have that inference on clusters somewhere, better yet, because then on VLLM or wherever that inference provider is, you can start to do even deeper analytics and understanding of some of these things because some of those third-party providers have actually started gating the thinking blocks and stuff. So if you need to get like every single piece and every single detail, I think it's very worth doing. The other things that I think, you know, if you're thinking, we don't need to go that far. We don't want the many gigabytes and gigabytes and gigabytes of our agents thinking about Kubernetes. Tool calls, I think, are an absolute must. So every single kubectl call that an agent would make, I think, is critical. MCP calls, which are tools themselves, I think also really worth capturing and then logging. And then I think the penultimate of doing this would actually be finding a way to lift and shift those agent operators into your cluster or into some infrastructure that you can actually apply some logging and tracing on top of as well. Because then that way you would have full end-to-end understanding and control of really the observability metrics or the observability levers on that thing. Just like any other piece of software or pod or something you would throw into a cluster or throw into some infrastructure, you're going to get the logs, you're going to get the traces, you're going to get like every tool call and everything on top of that. This starts to get into, you know, some things people have been talking about where it's like, oh, you should really like be building your own harness. You should really be building like all the stuff all the way down through the stack. I'm like, there is an argument for that. It's a lot of software to own and manage. You just have to ask yourself the operational question of okay, how much can we do for our own harnesses? How much can we even manage those harnesses with agents? And the AI software development lifecycle continues. But last thing I'll say on this is I also think the alerts and the metrics and the things that fire off automated incident reporting are very worth considering as a critical part of this whole loop of incident response. And that's not really new. Like I'm saying all things new become or all things old become new again because these are just good old-fashioned like observability signals. So make sure that you're looking at those alerts and what drives the agents because that's ultimately going to be a big part of what keeps them persistent. It's going to be a big part of how they continue to drive towards some goal when they get kicked off by some alert or something.
Bart Farrell: On the subject of incident response and alerting, as AI becomes a critical part of Kubernetes incident response, how should teams think about operational memory? And what information from past incidents is actually useful for an agent to learn from? And how do you prevent outdated or context-specific fixes from being reused in the wrong situation?
John McBride: I think this is a hot topic right now, because I don't know if anybody really knows exactly the right answer here. Maybe this is just the age of AI where it's a lot of vibes and we're just kind of like figuring it out. But I think that AI agent memory systems should more or less be purpose-built for that team or operator. If some team is running a bunch of EKS clusters, and they're already using Bedrock, it probably makes sense to figure out ways to integrate really deeply with the stacks in the software that you already have, instead of trying to bring on some very generic solution, you know, and I think you can go really wrong here really quickly, where you're trying to do a bunch of stuff and like an AGENTS.md or trying to like, capture every past decision or every truth that like emerges, but oftentimes as these things kind of evolve and you end up with like past decisions polluting like the current state of what is actually real. And really, this is just like good old-fashioned prompting, like good old-fashioned, how transformers work, inside an actual like model, because ultimately that attention and this goes back to all you need is attention, which is this classic paper on how general purpose transformers actually work. If you're giving an agent inside of a prompt or any kind of context, or like goes and reads a file and it sees a bunch of stuff, that'll pull on its attention. That'll pull on all the kind of weights and the peaks and valleys inside of the model to pull it a certain direction. So I think personally, I think it's worth being like hyper-surgical on what memory you keep inside of that whole like agentic loop essentially, or in the prompts or in the AGENTS.md or in whatever memory system you decide to utilize. I personally really like an Obsidian vault for this. And Obsidian actually has a great little CLI for search that agents can just use. And Obsidian also gives you kind of the whole link back for things, which basically just becomes like a little mini wiki. But oftentimes, I try to keep it about as clean as possible to just at least in my own personal use cases to just like the current state of the world and the current truth of what like is going on today, because these things just change so fast and so quickly. So, you know, there are some there are some arguments to be made that previous decisions or things that happened with like, you know, scaling or like, you know, even a short little blurb, like, hey, you know, two months ago, we saw this where like this crazy scaling thing happened, or we had to like defensively bring up a bunch of pods, a WAF, or some other network rules because we're being attacked. Do that again, if you see that, you know, that could be very critical context for a high-impact situation. So again, I go back to every team should think about agentic memory for what works for them. And what's worked for me in the past is just kind of bland markdown in the current state of the world. We'll see what emerges. I imagine this will be an area that AI labs are very interested in as they continue to push the horizon nature of these agents and how big of tasks and how long-form tasks they can actually work on.
Bart Farrell: And John, what's next for you?
John McBride: What's next for me is to keep working on the Paper Compute Company. Check it out at papercompute.com. We'll be at KubeCon. So, come say hi.
