Bart Farrell: First things first, which three emerging Kubernetes tools are you keeping an eye on?
Nadav Erell: Kro is one. Kro is interesting for me, because it touches some of the stuff that we're working on in Skyhook and then Radar of how you define, applications or groups of resources in Kubernetes and not treating everything separate things. An application is composed of a service, a deployment, replicas, pods and all the other different resources. And so packaging them in a more standard way, a tool that standardizes, is very interesting. And it's kind of been missing, I think, in the landscape. So excited to see if it gets significant adoption. The other two are unsurprisingly AI and agent related. So I would name K-Agent and Agent Sandbox. Kubernetes is obviously where most probably of agent deployments are going. And these are sort of two pieces that are key to this. There's more to building an agent than only what the framework that agent provides. But these are key pieces that you really want to have good, solid, well-supported, standardized foundations for how you run your agent runtime as a resource in Kubernetes. Agent sandbox, because you need the agent to be able to do useful things, be able to have a sandbox to operate in, to be able to use you know bash tools edit code, things like that, you really need to isolate from the environment. And containers is some boundary, but it's really not enough for what you actually want the agent to be able to do.
Bart Farrell: So when an AI agent investigates a failed Kubernetes deployment, what evidence should it retrieve first? And how should it correlate the desired state in Git, generated manifests, CI/CD logs, Argo CD sync status, Kubernetes events, live resource state, and application telemetry? And what happens when those sources disagree or contain stale information?
Nadav Erell: With agent investigations, they can collect a lot of evidence. As you said, they can look at the state in Git, they can look at logs, manifests, and so forth. It can be a lot. You do want them to fan out, and agents can be effective at gathering a lot of things. But you don't want to go overboard and collect irrelevant things that will just confuse the agent, burn the whole context window. So ideally you try to understand, to narrow down where the problem is likely to be or when, so you're not collecting for example, logs that go before when the problem actually started. And so you want to do it progressively and maybe start with what is the symptom that you're looking at. If there is something that you can identify directly from for example, the resource in Kubernetes. Great, you can look at that. You don't necessarily have to go and fetch deep logs if you get the clear error message that says something like, image pull error, something like that. It's very clear what the problem is, and you have to do almost no work to figure that out. So there's things that are easier to go first, like the status of the resources in Kubernetes, and then based on that, decide where to dig deeper. So the basics would be the status of the Kubernetes resources, the sync status in Argo CD and Kubernetes events, and then go deeper into logs and so forth because they can be very time-consuming and context-consuming. Now, when things disagree, that is part of the diagnosis in many cases. Like there's a reason these things are not seeing the same picture. It's pieces of the puzzle that you would want the human or the agent to kind of put together. So it's extremely interesting when you have something that looks conflicting, that hints that you're looking at something that deserves closer attention. And one more important thing is that to remember that you could be missing evidence. Maybe things are not collected properly. Maybe if something happened more than an hour ago and you're looking at Kubernetes resources, by default they expire after an hour. So it's important to also remember that lack of evidence needs to be interpreted carefully.
Bart Farrell: Suppose the agent has read-only access to the cluster, and any proposed change must go through a pull request and GitOps reconciliation. What should that pull request contain beyond a YAML diff? Which tests and policy checks should run before approval? And what can still go wrong between merging a correct-looking change and Argo CD applying it successfully?
Nadav Erell: A YAML diff is okay, but it's not what actually gets applied to the cluster. It still goes through some transformation. The YAML, you want to diff the rendered output; look at the render diff, not just the source code diff. So that's one critical thing, because that could surface something that you didn't expect, that you didn't anticipate happening. And then you want to include the evidence bundle of the things that you saw that were broken that this is trying to fix so that another agent or a human can look at it and say yes like this if it does it makes sense this tries to address this evidence that we're seeing and probably you also want to write down the expectation of what should happen like what should happen once this gets merged so for example this gets rolled out we're supposed to see this service this workload go healthy or not or whatever. So you have kind of no expectation and can verify it after the fact. In terms of what you should want to actually check, so the simple things are just the actual schema. In theory, you could merge YAML code that is literally broken and doesn't even render. And then the next one is you can render something that is valid but gets rejected by the Kubernetes API server when it actually tries to deploy. It's just a waste of time because you can do a dry run apply and get that answer first. So before you actually have to go through this. So these are things that are kind of basic to make sure that you're not just wasting time on this whole loop of having a PR reviewing it, deploying it and so forth, only to have it rejected and just have to go at it again. And things can obviously still go wrong, you know, you don't always know what you have to fix. So you'd probably want to collect all the evidence again. The same thing that you observed that led you to this PR, you want to look at them again, but things can break. You could have other sync issues, you could have webhooks that change what gets deployed. So there are a bunch of things, you know, Kubernetes scheduling that can reject it. So there are a lot of things that can still go wrong. So you just have to re-observe them again after that happens. So merging the PR is definitely not calling it a day. to revalidate everything after that.
Bart Farrell: How would you build an evaluation suite for a Kubernetes operations agent? Which failure scenarios would you include? And what would you measure beyond whether it eventually produced the right answer? Time to root cause, number of tool calls, context size, false diagnoses, unsafe remediation, reproducibility, and whether a deterministic check would have been more reliable than an LLM.
Nadav Erell: So this is a great question. We did this at Radar when We were trying to evaluate it. So basically, we didn't do a whole independent evaluation suite, but we looked at existing ones and modified them to be able to actually show how Radar's MCP server works for this compared to other MCP servers or plain kubectl. And we found that you just need a variety of common failure scenarios you can collect, any number of them, you can have scheduling issues, admission issues, crashes, probes, network issues, storage issues. So you want to have a good variety of different fault classes. You want to have a real cluster with some healthy things running in it and an injected fault that you can clearly control and know that you have something specific. You might also have distractors, like other things that are broken, that the agent should know to ignore or should be smart enough. And then you want to make sure that you can run it in a way that is reproducible while remembering that agents are not deterministic. And that's fine. So you shouldn't really expect to reproduce every single scenario. But when looking at a whole set of let's call it 50 or something different scenarios, you're going to expect to see broadly similar results. And if you run multiple times, you're going to get a better result. I think it's also important to not have everything as like a zero or a one. In many cases, what you get, especially when you're with agents, is an explanation, interpretation, and diagnosis. Not everything, I think, fits well into a binary, oh, did the agent fix it or not? I think that can do some of the work, but you really want to see if the agent understood the chain of events that led to the errors and things like that. So I think that having AI-based scoring for the agent explanation is useful. So you can have a numeric score as if from 0 to 100. And then when you aggregate across the different scenarios, now you're getting something that is useful. In terms of what you're comparing, again, you don't want to compare only pass or fail. You want to compare the score. But accuracy is not the whole story. So, for example, what we saw that we used SRE-Gym. We don't run it, we use it, so it was useful. And we saw that if you use just a general agent like Claude Sonnet 5 or something like that, and just give it kubectl and tell it, you know, go do the 50 scenarios that we said. In many cases, it can take 15, 20, 25 minutes to get to a diagnosis. And that is, you know, it might get it correct eventually. But when you have an incident, you really don't want to wait 20 minutes to get an answer. So definitely measuring how long it takes to get to the correct diagnosis, I think is critical. And you can also basically have like a histogram of like what percent of the scenarios you got right within, let's say a five minute budget of diagnosing the issue. You can also look at like tool calls and tokens. I think they're relevant, but I would call them secondary. I think the time to diagnosis is more critical alongside the actual accuracy.
Bart Farrell: What's next for you?
Nadav Erell: Next for me. We started this journey with Radar about six months ago as kind of a side project from what we're building in Skyhook. And it kind of blew up pretty quickly. And we realized a couple of months ago that there is so much more depth and such a need for what we're building at Radar that we're looking to really focus on making this the new visibility layer for humans and agents for Kubernetes. We started Radar focused less on AI and more on just giving people, humans, a good interface that is actually enjoyable to use to understand Kubernetes to make it more accessible. But as we were working on it, we realized, you know, as you can see that more and more people are using more agentic workflows, and that has become a bigger focus. So we're adding more and more capabilities to make Radar basically the best interface for agents for Kubernetes. So that's the focus for us.
Bart Farrell: If people want to get in touch with you, what's the best way to do that?
Nadav Erell: You can find me on LinkedIn. Nadav Erell, you can email me at nadav@skyhook.io. You can always also check out a repo that is open source on GitHub and we have a Discord, so whichever way you want. I would say that email or LinkedIn are probably easiest.
