AI Incident Response: From Reactive to Predictive

AI Incident Response: From Reactive to Predictive

Aug 31, 2026

Guest:

  • Sylvain Kalache

The future of incident response might use AI to help, but fixing problems safely still requires careful engineering.

Sylvain Kalache, Head of AI Labs at Rootly, explains how AI can make incident investigations faster, what information it needs to find problems, and how teams can balance automation with checks, approvals, and hands-on experience.

In this interview:

  • How AI collects and connects incident data

  • Why both computer data and human knowledge matter for solving problems

  • What rules are needed before AI can fix things

  • How we might start to predict and prevent incidents, not just react to them

Subscribe to KubeFM Weekly

Get the latest Kubernetes videos delivered to your inbox every week.

or subscribe via

Transcription

Bart Farrell: Sylvain, welcome to KubeFM. For folks who don't know you, who are you, what's your role, and where do you work?

Sylvain Kalache: Thanks for having me, Bart. My name is Sylvain. I'm the head of Rootly AI Labs. The role of the AI Labs is to explore what AI, and more particularly, LLMs can bring to the world of reliability. We do this by building a lot of prototypes, open source tools, research that we share to the community. We are partnering with companies like AWS and Google. Think of it as a constant hackathon, but where everything is shared with the community.

Bart Farrell: Now, where is AI genuinely improving incident response today? And where is it still creating more noise than value?

Sylvain Kalache: I work for an incident management company, and I see a lot of customers using AI to help them in this incident response. It helps with all the different steps of managing an incident. First, when something breaks, the first thing you need to do is collect data. You need to look at your system and integrate it to understand, look at the state and the logs and the traces and so on and so forth. That process is very time and energy consuming if you are doing it manually. You need to open a bunch of dashboards and maybe your terminal and spin up some command and whatnot. Collecting this data can take a few minutes up to maybe half an hour. And in the meantime, the clock is ticking for your incident. So here, AI is helping a lot into collecting everything you need in basically one interface. The MCP protocol made this very easy, with regular API requests. In a matter of a few minutes, AI can help you to collect all the data that you need to go to the next step. And the next step is kind of correlating the data. Using AI, you explore multiple hypotheses. Maybe you're like, oh, the database is slow, could be because of this, because of hypothesis A or B or C. AI can in parallel collect all the data and then here you will use the power of LLMs for it to look at it and try to find some correlation and potential root cause. Which leads me to the next step of an incident response step which is solving the outage. Here again, once you've found the root cause, LLM can help you to craft a draft of the solution depending on how complicated or how comfortable incident responders are, they may let the AI do it all by itself or at least guide the engineer in the direction that they may want to take. Finally, once the incident is solved and there is no more fire, there is a bunch of incident follow-up that needs to be done. One of them is called incident postmortems or incident report, where you need to document exactly what happened, what caused the issue, how this issue was investigated along with the timeline, how it was fixed, what was the customer impact and so on and so forth. This type of report is generally very long, time and energy consuming to do and here again AI can help you to craft this document, at least come up with a template to help you come up with it. In summary, while we've seen actually an increase in the number of incidents since GenAI started from 2023, which is when OpenAI first released their first model and people got excited about it, up to today, in 2026, we've seen the number of incidents increase, average number of incidents per customer increased by three, which is pretty big, but we are also seeing incident responders being able to solve this incident much quicker thanks to everything I just described.

Bart Farrell: What operational context does an AI system need before engineers should trust its diagnosis? Logs, metrics, traces, deploy history, previous incidents, ownership data, and how should that evidence be presented?

Sylvain Kalache: There are two types of context. Everything you just mentioned is a must-have: all information related to the system itself, like logs, metrics, traces, your deployment history, maybe former incident, and so on and so forth. But I think there is something else that needs to be shared with this tool, which is something that's generally more human-centric. Let me explain it. Maybe let me take a step back a little bit. If a human needs to investigate an incident, if they've been with the company long enough, or if they've been in their career long enough, when something breaks, they kind of have an instinct, this intuition of like, hey, when the service is misbehaving, generally it's like, this part of the infrastructure, or like, hey, we are having an issue with a vendor or whatnot. They kind of already know where it is, and so the goal is to empower this AI-assisted incident response tool to have the same intuition. And while the log that all of this is good, sometimes, there's more than this. Here, stuff like documentation, cookbooks, maybe having access to your Slack conversation, or maybe even if you are recording your meetings with your coworkers. You know, there is a lot of things that we as humans will say, at the coffee machine or in the open space. This knowledge that's not code, but that might explain the business reason about why this service or this piece of code has been designed this way. Maybe why there is a spike in traffic at this time. All of this is like knowledge and information that can empower this tool to basically provide better and faster results.

Bart Farrell: Where should teams draw the boundary between AI explaining, recommending and taking action during an incident? And what approvals, audit trails, and evaluations are needed before moving toward remediation?

Sylvain Kalache: Whether a code or a fix is typed by a human finger on a keyboard or generated by another piece of code, at the end of the day, it's not that important. The same level of rigor should be applied to human-generated solution or machine-generated solution. That means that when we, as SREs, want to push a fix, generally, it's maybe reviewed by a coworker or going through a CI-CD pipeline. The same should apply to a fix that's pushed to an LLM. It's amazing to let this system end all incidents for us. Because solving this incident is often stressful and time-consuming. Engineers don't like that. Sometimes you have to do it at night, or maybe you have your family or friends. And having machine-solving problems is definitely something we've been, as human engineers, aiming to do. But one risk with that is you may be losing touch with your system. If you let this AI-assisted tool, this AI tool, do the work for you when solving all of these incidents, you never deal with them someday like a very large, complex incident. You might come and you are kind of lost because you've kind of forgotten how your system works and you didn't develop this mechanism and instinct I spoke about earlier in the interview. So I think there is a fine line between using this tool and putting them on autopilot and having the approval of like, hey, just do whatever you want and keeping the human in the loop, which is a term that was popular a few years ago, but I think it's still relevant and make sure that the engineer still knows what's up and can go with automated or do it manually.

Bart Farrell: And Sylvain, what's next for you?

Sylvain Kalache: What we are working on at Rootly is exactly what we just discussed, AI assisted incident response. Now this approach is very reactive. Something breaks, you try to fix it. Engineering teams don't want things to break. What we are working on is to not only be reactive, but to be proactive and even predictive. Now with the rise of AI, and machine learning, we have capabilities to be able to predict, or at least warn humans of the process that something may be going wrong if you move forward. What's next is that we are trying to address problem management and to address problems before they become incidents.

Bart Farrell: And if people want to get in touch with you, what's the best way to do that?

Sylvain Kalache: I'm big on LinkedIn. I post a lot of memes about incident management and AI. That's the best way to find me. Also, I have a website, sylvainkalache.com.

Subscribe to KubeFM Weekly

Get the latest Kubernetes videos delivered to your inbox every week.

or subscribe via