Bart Farrell: Welcome to KubeFM. For folks who don't know you, who are you, what's your role, and where do you work?
Kaustubh Phatak: Hi, my name is Kaustubh. I work for Crusoe. I'm a senior staff software engineer working on the managed Kubernetes platform, especially on the GPU data plane side. I own the GPU stack at Crusoe when it comes to Kubernetes.
Bart Farrell: In a managed Kubernetes platform, how do you define clear responsibility boundaries between the platform team and application teams without creating blind spots during incidents?
Kaustubh Phatak: This is a very classic example of who owns what. The platform team usually owns the actual control plane, node lifecycle, a ton of other capabilities: OS, drivers, (if you run virtualized workloads) the virtualization platform, the GPU operator, the network operators, and the health checking. The app team actually owns the workload config, resource requests, app level health checks, retries, checkpointing for their app. But where the incidents actually lie is on the boundaries of these. To fix that, what we do is we define the boundaries with SLOs and not with the org charts. The platform publishes the commitments like API schema or API availability metrics and the node replacement time, the order remediation, labels, annotations, which are required. The blind spots that appear near the seams of these teams, we try to not be bounded by that. For example, in a NCCL timeout, if you're running it on thousands of nodes, we have to figure out if it's an app config issue or a fabric issue or a NCCL parameter being wrong. We give both the teams, the platform team and the application team, a view of the same metrics. Their agents and human counterparts can look at that data and figure out who owns the incident. By default, the triage is always by both teams until we know it's a problem with an app or the platform. Where AI actually helps with this is it has a view of all. It can look at app logs, try to correlate with the platform or what is on the hardware, what are the errors we saw on the GPU hardware, and it can correlate very fast versus the human counterpart. For the incidents that happened right on the boundary between the teams, once an AI agent can act on both the sides, the first question is how to solve it, not who owns it.
Bart Farrell: What operational evidence should a Kubernetes platform preserve after a configuration change, failure, or automated recovery so that an incident can be reconstructed reliably afterwards?
Kaustubh Phatak: I would say it's not any different than what a human SRE or a DevOps person should preserve. In short, it's who, what, when, why: the four Ws for every change. And we should make sure that the evidence is not lost if something is deleted, or a node is rebooted or reformed or whatever. What to preserve particularly is change records, including API server audit logs, or if you use GitOps, use the GitOps history for the desired change that went in. For automated logs, preserve triggering signals and correlation IDs. The outcomes of every automated action should be logged and preserved. We should also preserve node conditions, labels, driver information, cumulative versions. Nothing should be unknown or pulled from upstream all the time. It should be reproducible no matter what. We should also have metrics, logs, hardware errors, all of that. And where AI actually helps with this is it knows where to look for things. And all we have to do is add a new source so it can go fetch it. And it can generate a nicer, cool-looking report, presentations, dashboards. We use AI to generate Grafana dashboards. And it does it on a whim. And it makes the evidence usable. And it makes the mean time to resolve actually very low. It's a huge advantage.
Bart Farrell: And how do you design automated recovery so that the platform can heal quickly while still keeping changes controlled, observable, and auditable?
Kaustubh Phatak: I would say it depends on how the platform is structured. At Crusoe, we took a different approach to remediation. We try to remediate in an automated way where we gather metrics from GPUs using open-source tools or homegrown tools. We are there to our platform to change the reservation, use the Kubernetes constructs to drain and cordon the node and send it for repair or GPU reset or all of that. In general, the design should look like: remediation needs to be in the code and not manually done. It needs to be triggered by a signal and it needs to be very bounded, limiting how many nodes can be remediated at any given time, with some circuit breakers and rate limits. And it actually helps with this quite a lot because, again, it has all the views of the system. It knows the Kubernetes constructs very well. It can look at PDBs; it can look at labels, all the taints, tolerations. It has knowledge about the pre-stop hooks. We can utilize all of that. And I would say we shouldn't trust AI to do actual remediation. The human needs to be in the loop for any approval actions or any actions that are not reversible and that destroy the state. AI should make the recovery faster to diagnose, but it needs to be audited as well. Everything it proposes goes through the same guardrails and leaves the same trail as any human would do.
Bart Farrell: Thanks so much for sharing your knowledge and experience with our community today. Look forward to hearing about the next steps. And best of luck. Take care.
Kaustubh Phatak: Thank you.
