Bart Farrell: Sylvain, great to have you with us. For people who don't know you, who are you? What do you do? Where do you work?
Sylvain Kalache: My name is Sylvain. I currently live in Florida. I run the AI lab for Rootly. It is an on-call incident management platform used by companies like LinkedIn and NVIDIA. And my role at Rootly with my team is to explore what AI can bring to the world of reliability. We build a lot of prototypes, open source tools. We also do a lot of research that we share with the community. Everything we do is open source.
Bart Farrell: Now, we want to dive into this topic of bank-grade Kubernetes, so we've got some questions for you. A company called Wealthsimple expanded incident command to a pool of roughly 60 to 70 people by embedding guidance into the response workflow. In a Kubernetes incident, which steps should be standardized so that less experienced commanders can respond safely? And which decisions still require deep technical judgment? For context, Wealthsimple is a customer group.
Sylvain Kalache: I would break down my answer into two sections. I would say that one of the first steps when you are responding to an incident is to collect data. And this step should obviously be automated as much as possible. whether by having an existing internal developer platform with a bunch of dashboards that will show you the key metrics that you need to know. If ever this is not done, what are the requests that you need to run against your database or datastore to get answers in the form of cookbooks? So that's, I would say, the first step that needs to be standardized. The second one is the next step, which is the collect data, obviously you find the root cause, and now you need to perform an action. Whatever it is for Kubernetes cluster, it might be restarting a cluster or the service, reversing a recent change, or removing, draining a problematic node and whatnot. And ideally here, you don't want maybe to let incident responder have to use a raw kubectl command, but you might just want them to push a button, right? Like maybe it's through Slack or maybe through your IDP or some click in some dashboard so that you ensure that they can do it safely with the proper guardrail and really minimize any potential human mistake or any sort of mistake. Decisions that still require deep technical judgment is everything that's going to be out of the standardized processes, whether that's an investigation or a resolution step. It might be a service where there is no graph, nothing on the dashboard, and you need to manually query them. Or maybe none of the cookbook suggestions on how to solve this problem are working or seem to address the system. the issue, the problem that the incident responders are facing. And so in this case, you might have to do custom things that are not properly protected by guardrails. And so in this case, you still need this deep technical judgment that standardization is not always providing.
Bart Farrell: Another company called Upstart runs Kubernetes while serving a platform used by nearly 100 banks and credit unions. And they wanted to automate their existing incident process without imposing a rigid, one-size-fits-all model. So how exactly do you encode incident workflows without creating runbooks that break as soon as the failure is unfamiliar?
Sylvain Kalache: This has been an issue for a long time. For instance, back when I was an SRE at LinkedIn, we had this software called Nerf, which was this automated incident response tool where you could trigger cookbooks to solve incidents. Sometimes there are no cookbooks that would match what you were adding and you had to go the manual way. Since the last few years, LLMs made a huge gap in bridging this issue by bringing reasoning, right? So if you face an incident where the solution is not encoded, you can empower your tool to use LLM to reason and look at the state of the system, the data you have available, and at the very least, it's going to be able to come up with ideas of what may be going wrong. Nowadays, we are seeing this system with very high accuracy, finding what's going on and even coming up with a fix. So this dilemma, I think, is no longer an issue. And that's a very powerful workflow that companies can embed into their incident response process.
Bart Farrell: For financial organizations such as Brex, SoFi, Wise, and Wealthsimple, what evidence should be captured automatically during a Kubernetes incident so the response can be reconstructed later? But without slowing down the engineers that are trying to restore the service.
Sylvain Kalache: Data retention is a big topic. Ideally, you need more than ever your code, and you need to be instrumented. You need to have the right data collection in place prior to an incident happening, so that, to the point of the question, engineers don't even need to think about that. So that's really something infra team needs to think about ahead of time. Now, obviously, my opinion is that the more data, the better, right? The more context, the better. And the reason is that the company you just mentioned here, Brex, SoFi, Wise, Wealthsimple, are running extremely complicated and sophisticated infrastructure. And sometimes the difference, the reason why their system might not be working well might be something like very minor or very subtle. Maybe it's like a latency that's a few hundred milliseconds above what it needs to be, right? And that's disrupting the entire system. And so teams who respond to an incident might be able to mitigate the issue fairly quickly. But ultimately, sometimes finding the actual root cause might be like finding a needle in a haystack. And in this case, you will need all the context that you may need. And this type of investigation can take days and sometimes weeks. To summarize, I think the more context the better. And then you can just keep the full context for a limited amount of time. And let's say then you can rotate after a week, maybe you can ditch most of them and so on and so forth.
Bart Farrell: So, Sylvain, what's next for you?
Sylvain Kalache: So, at Rootly, we've been historically building tools to empower SRE teams to manage and respond to incidents as fast and as well as possible. We recently released an AI SRE, that's an AI-assisted incident response tool to, again, be as fast as possible to solve incidents. But now we are thinking to shift left and use all the intelligence that we got from helping hundreds of thousands of incidents to be solved on our platform to empower and come up with tools that provide proactive and predictive capabilities to our customers so that they can solve a problem before it becomes an incident.
Bart Farrell: And if people want to get in touch with you, what's the best way to do that?
Sylvain Kalache: Two ways. Being on LinkedIn, I like to create funny memes about infrastructure, incident management, and AI. I'm having a lot of fun with that. I have a website, sylvainkalache.com, where you can find more. I actually have an MCP server, so you can play with that.
