Bart Farrell: First things first, who are you, what's your role, and where do you work?
Kaslin Fields: I'm Kaslin Fields, and I lead GKE Developer Relations Engineering at Google Cloud.
Bart Farrell: Ray and Kubernetes both have their own schedulers and resource management models. When a distributed AI workload gets stuck waiting for resources, who's actually responsible? Are we making troubleshooting unnecessarily complicated?
Kaslin Fields: Ray was designed as a solution for distributed machine learning use cases. Kubernetes was designed as a solution for distributed compute, especially for web-server-type, long-running workloads. The schedulers work a bit differently. And the way that Ray abstracts the underlying hardware is a little bit different from the way that Kubernetes does. Ray uses Kubernetes; they work great together because Kubernetes has such a long history of doing distributed compute. Ray uses Kubernetes as the distributed compute provider layer, and then abstracts it away again and provides a nicer interface for data scientists or folks who are used to building machine learning workloads and not having to worry about which type of resources are underneath. Kubernetes provides the labeling: here are the resources that you have in your cluster. And Ray does some of the scheduling, abstracting that away and making it really smooth for machine learning workloads.
Bart Farrell: KubeRay is supposed to bridge these two worlds. What does it actually solve for platform teams? And where do difficult problems around GPU utilization, scaling and reliability still exist?
Kaslin Fields: The teams that I see using KubeRay or Ray the most are platform teams that are serving data scientist teams. These data scientist teams are building models, they're doing training, they may be doing inference as well, fine-tuning and reinforcement learning, all the things involved with building a great model. Ray is a great solution for that. And then platform engineering teams are the teams that solve the infrastructure problems. Data scientists don't want to worry about what kind of hardware is underneath. The platform engineering teams have a role in setting up what the policies are, what clouds those resources actually exist in. Platform engineering teams are the ones that I typically see putting KubeRay on the cluster and then providing that as an interface to their data scientist teams.
Bart Farrell: If a Kubernetes platform team wanted to experiment with KubeRay tomorrow, what workload would you recommend starting with? And what's the biggest mistake they should avoid before moving into production?
Kaslin Fields: I would recommend starting with an inference workload. There's a wide variety of different styles of models and the way that models are implemented and how they actually run on the underlying accelerators. If you want to start with KubeRay and explore what KubeRay provides you, not dive into the model and that inference stack, but how it runs, the platform engineering concerns underneath it, I would pick a well-paved path: a model that has a lot of tutorials already, or an inference stack that has a lot of tutorials already on how to do it on KubeRay. Follow that well-paved path first so that you can see how to install KubeRay on the cluster and what services it provides for that well-known inference use case.
Bart Farrell: And in 30 seconds, Kaslin, what's the one challenge that you'd like to see more effort and resources going towards in our community?
Kaslin Fields: I'm really excited about Agent Substrate. Before the pandemic, even before the current round of AI, I heard from AI/ML communities, HPC communities all the time that the Kubernetes scheduler wasn't great at AI. And we in Kubernetes have been changing that with Dynamic Resource Allocation, with llm-d as a separate project, with Agent Substrate as a separate project. There are a lot of new projects creating the new path for how we do distributed compute for AI workloads today. The biggest problem that I want to see us solve is making that path really easy for users. I think there are a number of efforts, which makes it more complicated. But the easier we can make it for folks to understand how you do distributed compute for AI workloads and what we mean by AI workloads, the better things will be.