Bart Farrell: Welcome to KubeFM. For folks who don't know you, who are you, what's your role, and where do you work?
Kaustubh Phatak: Hi, my name is Kaustubh Phatak. I work at Crusoe. I'm a senior staff software engineer working on the managed orchestration team, especially the GPU data plane of the managed Kubernetes product.
Bart Farrell: When distributed training fails, how do you distinguish a Kubernetes problem from a GPU, networking, or storage failure? And what signals actually matter?
Kaustubh Phatak: That's a good question. In general, to debug a Kubernetes problem, usually we go through certain basic criteria where the pod is Pending, where the container is in ContainerCreating state or it's in CrashLoopBackOff. Those are general Kubernetes signals. We also go as a human debugging these things. But with AI in the picture, AI helps us build the timeline of what fails, because if we provide the AI with all sorts of logging metrics, all the data sources we have access to, it can correlate that timeline with GPU telemetry, CPU, GPU counters, fabric counters, all of that. And it can find the failure that happened first and not later in the stack, which usually bubbles up as the application error or, for example, in an NCCL failure with thousands of nodes, it shows that the distributed training failed and certain thought field, but we don't know what node it actually failed on. With the help of AI, we can ask the agents or LLMs to go find all of these logs and correlate those with metrics. Where it actually helps is figuring out what those errors mean. For example, in NVIDIA's case, it exhibits Xid errors when NCCL fails on a distributed training. That usually points to some error, which is either on the hardware or on the application side. And AI can translate that based on the previous information it has, previous incidents that happened, or online documentation from NVIDIA. As a user, it will take hours to actually go through all of the documentation, figure it out, correlate, convert UTC time to your time zones (it's always confusing) and come up with a ranked hypothesis. But AI does it in a very short time and in a very predictable manner. I wouldn't say it's always correct, but it does help narrow the problem and get to the shortest mean time to reproduce or also to resolve.
Bart Farrell: Which failure recovery decisions can safely be automated? And where should humans stay in the loop?
Kaustubh Phatak: With the improvement in AI, the natural tendency is to automate everything, give every control to AI. But in practice, any deterministic failure signatures or any previous known issues can be automated very safely. Even we can automate it using AI. For example, some Xid errors where the vendor recommends replacing the node or resetting the GPU or restarting the GPU. Those all can be very easily automated. Any stateless workloads in general in Kubernetes, if they don't have PVC or PV, can easily be automated. Specifically in distributed training, swapping a node when you have a known checkpoint and then restarting the training job is also very automatable. It's a little harder to automate, but it's still doable. Our customers do it all the time. When it's not safe is when you see a pattern or a correlation where there are multiple nodes failing or it suggests a fabric-wide issue, and those things should not be remediated, should not be automated because that can cause cascading failures across the fleet. At Crusoe, we do multiple levels of circuit breakers, rate limiting, cooldowns for some node remediations. And to answer the last point where the human comes into the picture is actually at every stage. Even in the automated way, the human needs to look at the previous logs, what was automated and how it was done. What we cannot automate is something that destroys the estate. For example, database deletes. We have all heard stories about some agent going and deleting a database. Those things should never be automated. And where AI helps particularly is it can spark the correlations. 40 nodes failing might be connected to the same switch or the same rack or the same firmware version. And it can go figure that out very quickly if we have a set of instructions given to the agents. And another point is AI shouldn't be the executor. The models are not deterministic. If you ask the same question to a model five days apart or even two days apart, it can give you different answers. The agents can only recommend, but all the automation or remediation should be done in code, which is very predictable and testable.
Bart Farrell: And what evidence should an AI agent have before you trust it to recommend or take action in a GPU cluster?
Kaustubh Phatak: That's a good question too. We have all seen that SRE agents are being automated. There are a lot of companies standing up for this. I would say we should hold the AI agent to the same bar as any new SRE: that it needs to show why a particular recommendation needs to be done. Specifically, what signals it saw with timestamps, the links to the metrics, which can be verified within 30 seconds. A user doesn't have to go through hundreds of lines of logs every time. We also want to see what it ruled out and why, because the AI agents only know what we have exposed that agent to. In the sandbox environment, if you don't provide certain logs, the error might just land there, and the agent will try to come up with a hypothesis which might be wrong. The AI agents also need to know what similar past incidents have happened, so they can correlate and produce that evidence. At the bottom, we don't want its confidence, but we want evidence-based recommendations. Where this really stands is if we allow the AI agents or the SRE agents in some shadow mode where they can look at previous incidents, stay with the on-call maybe alerts or Slack for weeks before we allow them to recommend or remediate anything. And another important point is these AI agents need to also maintain an audit trail of what they have done, so a human can go back or another AI agent can go back and figure out what was done by the agent instead of keeping it loose in the wild.
Bart Farrell: Excellent. Thank you so much for sharing your experience with us today. Look forward to checking these opinions out more in the future and how the community will react. Take care.
Kaustubh Phatak: Thank you for having me.
