Salman Iqbal: Bart, explain KubeSelect to me in five seconds.
Bart Farrell: Five. One bold Kubernetes hypothesis.
Salman Iqbal: Four.
Bart Farrell: We test it with an expert.
Salman Iqbal: Three.
Bart Farrell: Evidence, experience, no hot takes.
Salman Iqbal: Two.
Bart Farrell: And see if it survives.
Salman Iqbal: Not bad, Bart, not bad.
Bart Farrell: Thank you, man.
Salman Iqbal: Welcome, everybody, to the brand new show from KubeFM. We're calling it KubeSelect, the show where we take one bold hypothesis around Kubernetes and put it to the test. No hot takes. With our guest, we're going to look at this hypothesis and see if it stands the test of time. I'm Salman. I'm joined by my co-host Bart Farrell, also known as the KubeFM Podfather, because he's been making Kubernetes podcasts long enough to know where all the bodies are buried. And he's also made me an offer I couldn't refuse to co-host this podcast. Welcome, Bart. How are you feeling?
Bart Farrell: I'm feeling great. That's the best introduction I've ever received. This is a new thing for me being called the Podfather. And I feel like I have a lot to live up to, which is why we have an excellent guest who I think you're going to introduce now, Salman. Take it away.
Salman Iqbal: Absolutely. Our guest today is none other than Henrik Rexed. If you spend time around Kubernetes, learning around observability, chances are, you have seen his stuff online. He is Senior Staff Engineer at Dynatrace, a CNCF Ambassador, and most recently, an AI Foundation, an Agentic AI Foundation Ambassador, and the creator of one of my favorite shows on YouTube, Is It Observable? He shares practical information and hands-on content covering everything from OpenTelemetry, Kubernetes, platform engineering, and a bunch of other stuff. He's also a regular conference speaker. So if you had a conference, you will bump into him. And he's a passionate advocate for cloud native community. Henrik, welcome to the show. How you doing?
Henrik Rexed: I'm excellent. I'm very happy to be here, especially with a Podfather. That's impressive.
Salman Iqbal: Excellent. Thank you so much, Henrik, for your participation. So the purpose of the show is we pick a claim. And the idea is with the guests together, Bart and myself, we'll be testing this claim. At the end of the episode, we will see if the claim stays, or we chuck it out and put a different claim. And today's claim is this. Most organizations don't have an observability problem. They have a decision-making problem. We collect a bunch of metrics. We collect a bunch of logs, traces, OpenTelemetry, Prometheus, Grafana, you name it. The modern engineering teams collect enormous amounts of telemetry. Yet, at times, incidents take too long to resolve. Root causes remain difficult to identify, and engineers still lack confidence during production outages. I'm one of them, so I'm just putting myself to the test here. So the point here is observability really the problem, or are we struggling to make the decisions with the data that we already have? And throughout the episode, Henrik and I and Bart, we're going to discuss this claim, and he is definitely the right person to look to answer these questions. So let's start with the beginning part then. So the point, the question here is the claim. The claim is this, Henrik, the problem isn't collecting telemetry. It's knowing what to do with it. So the question, the opening question for you, Henrik, is when you look at organizations, you work with a bunch of organizations at Dynatrace and before, do they actually have an observability problem or do they have a decision-making problem?
Henrik Rexed: It's an excellent question, to be honest. And I think the main, I would say, I would derive it as slightly under-answered because I think in general, because I used to do a lot of performance engineering and load testing and everything, From the moment you are dealing with the systems, they will probably face problems because as a performance, you're breaking the app and you want to understand why. Then at the end, it's just about what's behind the app and what are the KPIs? What are the things that make me understand where is the potential problem located? So the reaction is, let's grab everything. So infrastructure, network, application. proxy, everything. So then you have a full picture. That's great. But at the end, you have to figure out where the data will flow in when the user or an external system interacts with the system. If you have that understanding, then you have a logical path where you go, oh, I'm going to check this. All right, doesn't work. I'm going to go in another layer, another layer, another layer, and then I'm going to locate the problems. I think it's more about, not about a decision problem, it's about how much do you understand your actual environment more than anything else. Because I think data, if you don't have the data, then you say, oh, I don't know. So you want to know. But if you don't understand, then you cannot take the actual decision. And that's my main answer.
Salman Iqbal: Fair enough. So what you're saying is the decision making is not the problem, but the context perhaps to make those decisions is the problem. Is that what we're saying, Henrik?
Henrik Rexed: Yes, exactly.
Salman Iqbal: So we've gone past this problem of trying to collect the data. I think you're saying it's kind of easy for us to collect all this stuff now, but we don't have the context. Is that correct?
Henrik Rexed: I mean, you said it's easy to collect, that's true. But collecting without a method and a process, you will collect a lot of data that is without the right context. So then you get a bunch of metrics, but you don't know where those metrics belong to or the relationship with a given component. And that's where context is super crucially important. That's why usually what we usually are talking when you are dealing with observability, you say, data enrichment. And data enrichment is all about, I have some data, I'm going to put extra information on top of the data. So then I can basically, when it will hit my backend or my observability backend or my storage, I have the keys, I have the options to basically query them and make the relationship between different components and different element systems.
Salman Iqbal: And then what do you see when you work with organizations in the past in order to help the teams to understand the context. of the stuff that we're saying is missing. What are the things that teams can do to improve the context? When it comes to, say, you wake up at 3 a.m., now you need to figure out that I need to solve this issue, bring the service back up. What is the thing that the teams can do themselves to better improve the chances of solving the issue quickly?
Henrik Rexed: I think there's been a very clear method that is very old now. It's simply the SRE methodology from Google. If every single service or component or whatever in place has already defined what are the key KPIs to measure the actual health, the readiness, their performance. there's plenty of them. But if you have those KPIs already pre-built in the systems and then the data that is collected has the right context on top of that, then you can go through those. Because when you have a problem, usually the first reaction is to say, hey, what about the SLOs? How are we on the SLOs? And SLOs, behind the scenes, there's an SLI, the metrics. So then you drive this. And then if you have this architecture of the dependency of the component, you fire the problem here because it's located here, and then you try to figure out if the problem is a real problem or it's a consequence of another problem that we may have not been alerted on. So you walk through this tree and figure out where it is. And I think the main problem is more than just the data. Because if I am paged out at the middle of the night, and it's a component that I have no expertise on, then I want to figure out who is owning this. So the notion of who owns that particular component, part of the broader applications. So then I have a team, I have someone that I can rely on, so then he can bring this expertise in the troubleshooting.
Bart Farrell: So we could say that lack of ownership or clarity around ownership is where engineers genuinely lose time because of trying to figure out who's in charge of this?
Henrik Rexed: I think there are a lot of systems, commercial systems for SRE perspective that brings that ownership. So once these it hits the systems, the ownership is defined. But I think in general, if you look at Kubernetes metadata, you have this notion of this component is part of whatever, so you can say that's the app. But this notion of ownership is not a standardized metadata yet. I don't think so. usually, in my case with Dynatrace, what we use, we have a specific label or annotation that we add. So this is the team ID whatever. And then based on what has been defined, I know that TeamID has different contacts for security, for performance, for whatever. And depending on the issue, you know who is the right channel or contact or Jira project I need to interact with to involve the right engineers.
Salman Iqbal: All right. Excellent. let me go back to the claim. And the claim was that we don't really have an observability problem, but we have a decision-making problem. And Henrik has answered really well in here saying, oh, we can collect the data, but we lack the context that allows us to make the decisions quickly. if you have the context, we can make the decision quickly. That's fair enough. So we'll come to that in a second. So let's move on to the next part. And you're collecting all this data, you know, everybody, Henrik's stuff is tagged correctly, everything is happening. Let's just look at, from your experience as well, what actually happens during production incidents. Because every engineering organization seems to have dashboards. Some have hundreds. But when production breaks, engineers aren't really looking for another dashboard. They're trying to answer the question. They're looking at what changed. Where did this problem actually start? Can I trust what I'm seeing? So the question for you is, Henrik, walk me through what actually happens when an engineer gets paged at 2 a.m. in the morning. Where does the observability genuinely help? And where does it stop being helpful?
Henrik Rexed: So to answer this, I said people usually align observability to only infrastructure, application, networking, proxy, and so on. But you mentioned I'm doing a commit. I'm in a push. I have a PR. I have an idea. I have something done through a CI/CD pipeline. So this is an event that should be sent. So then when there's a deployment event with a specific version number and everything, then if you see the actual problems, I look at, let's say, I'm a support engineer, I will start with a dashboard, but I would love to see the actual event. Oh, okay, at 2 a.m. before the incident, this has been released. That gives me at least the right context to say, okay, so maybe there's a correlation with this. And then if I'm technical enough, I will look at what was done in the PR or what was being tested. What were the things? And if I have access to this, I can confirm my hypothesis or not. But I think the biggest mismatch sometimes is the lack of connection between what we're doing on a CI/CD perspective, what we're doing in the code perspective, and then what lands in production itself.
Salman Iqbal: So is this where you think where engineers lose most of their time, just trying to figure out that thread of where things actually started? What actually happened? What led to this problem? Where do you see people losing time early in the morning when they're trying to figure out what happened? Is that where you think people are losing most of their time?
Henrik Rexed: It really depends, I would say, on the actual source of the problem. So it could be just an application change. So here, tracking what was changed and what was being deployed will help you to keep track on this. But then if it's an infrastructure, Kubernetes, that say, oh, we lost etcd, or we have a proxy problem, or there's a third-party DNS that is failing and that's causing this. So here you're depending on other data sources. So it really depends on the actual source of the problem. But I think that if it's an app related, it's very easy to keep track on. If you have the list of change mapped to your observed data, then the correlation between both of them will be very easy. Then if it's a DNS, then it's something else. So you'll have to check what you have internally. And then if you have the data from the DNS provider, then you can say, okay, confirmed, it's a DNS issue. So it really depends on the source of the issue.
Bart Farrell: And another thing that can become a challenge there, Henrik, when looking at observability, in previous years, and I think still today in 2026, we hear a lot about tooling sprawl. I know Salman previously mentioned dashboards for dashboards and it's kind of endless amount. Is that still something that a lot of SREs are struggling with in terms of when they're identifying what's going wrong is, like you said, where is this failing and the amount of things that they have to be keeping track of? in order to see what's going wrong. Do you think that tooling sprawl is still a significant issue in 2026?
Henrik Rexed: So I think now, in 2026, first of all, I think AI plays a role a lot. That will help the troubleshoot.
Bart Farrell: That was inevitable. I was going to also ask about AI SRE, to what extent that might be helping or hurting.
Henrik Rexed: I think in general, if you look at how we're dealing with our platform, we have this notion of problems. and then automatically through our system, we will pre-analyze dependencies and root cause. So then when you have a problem, you look at this, you have at least some numbers, and you can try to figure out, okay, I know that there's dependency, it seems that the dependency is the main problem. And then we introduced this notion of like a playbook where engineers define your SLO, SLI, and then they have predefined queries. So if you can query your observability backend, for example, I'm going to create some specific logs because I know that there's some log patterns that could be relevant. If this has been described and I'm an SRE being paged and I get this runbook where I know exactly the things that I need to play as a query to confirm one hypothesis to the other, then it's super helpful. But again, we're in 2026. We are jumping. And with AI, everything goes super fast. So why do I need to run the query where I could have an agent that is alert about the problem and because he has the actual troubleshooting process in mind or described, then he can run the query. So then I'm paged and already have like a sort of, or even an issue with all the details. Oh, I've been running these queries with these results. and here is my conclusion. And then as an expert, because I will be involved, I will look at this thing and say, oh, he's completely true, or in the opposite, oh, he went to the wrong directions. But at least trying to do the basic steps of troubleshooting analyzing done automatically and feed the systems with the actual results, I think we speed up the actual analysis.
Salman Iqbal: That's fair enough. That's cool. Well, let me ask you a question, Henrik. before we move on to the next part. How many times have you been woken up by a pager at 2 a.m. in the morning? Has that happened a lot to you or does it happen anymore?
Henrik Rexed: No, it doesn't happen anymore. As a DevRel, I've not been doing in that direction. I've been in my early career as a developer or in charge of stuff. You're stressed out at 3 a.m., someone calls you and then you're the expert. you're guiding and you try to resolve it. And then you have the pressure of the organization because they want to have you fixed because they're thinking of money behind the scenes. And when money is involved, usually pressure gets bigger. So I would say that if I had the tools that we have today 10 years ago, oh my gosh, it would be so much better.
Salman Iqbal: I'll tell you this a few years ago. We're talking about 2020 here. I got woken up and there was an issue that I had to figure out and it ended up being an issue in Kubernetes because of taints and tolerations. Things got kicked out. The taints and tolerations weren't matching up. And at 2 a.m. in the morning, you try and figure out in Kubernetes because you can tag things in a special way and why things are not getting deployed. From that day onwards, I started hating taints and tolerations. It took me two hours to figure out why it didn't happen. So we do some training with LearnKube as well. Anytime somebody asks me, should I use taints and tolerations? Well, when we start the topic, I explain what it is, but I said, don't use it. If you don't want to wake up at 2 a.m. in the morning and be a bit confused as to what's going on.
Henrik Rexed: And you can have the same thing with node selectors. I mean, if you have dedicated nodes for specific things and those nodes sometimes have the problem, it's the same thing. But I think it's useful because it helps you in how you want to ship things. But sometimes it's you shooting yourself in the foot as well.
Salman Iqbal: Absolutely. By the way, I'm not hating on taints and tolerations. It's a great tool. It's just, it woke me up at 2 a.m. and it puzzled me for like an hour and a half. I was tired. Thanks for that, Henrik. So let me move on a little bit more, right? So yes, we got all the data. We got all the, all the context, right? So hopefully all the things that you said, we're following. So we've got the context. So, here's the thing that most engineers aren't afraid of missing metrics because they're afraid of making the wrong decisions. Me at 2 a.m. in the morning. Remember that same issue that I'm talking about. So then imagine if they roll back too early or they restart the wrong service or follow the wrong alert? Right. Maybe observability isn't really about visibility. It's maybe more a bit more about a bit of confidence. So here's a question. What actually helps engineers make better production decisions? You touched upon it a little bit earlier on, like with some context. But let's just say, what do you think in today's world helps engineers make better production decisions when incidents are happening?
Henrik Rexed: So if you look at how we build software today, which is with agents, and then if agent has the right skills to instrument and to build things, when I finish usually building components, I say, okay, so you know the code, you know the app, you know what's part of it. We build specific metrics, we structure our logs in the same way. Okay, extract me all the right KPIs that allows me to report the health of these situations. We did that in the past, but sometimes we had to either be involved with a dev team or learn by experience. And now, because we're building it and it will go so fast, we can let them define things and that will help to basically troubleshoot. And then you can even ask them to write the right skills for this component or something that helps troubleshooting agent that will hit the actual data. I think there is a lot of people collecting a massive amount of data, and people are complaining about the costs in general of observability. I think the question is that if I'm collecting everything, yes, I want to be confident, I want to be sure. It's better to have lots than nothing. Sometimes lots is the opposite of being efficient. Sometimes we tend to collect too many things, and then if you collect too many things and you don't know the actual value of each single element that you're collecting, then yes, you have metrics, but then you look at the data and say, well, this is going up, but what does it mean? And if you don't know, if you don't answer the question, what does it mean, then what's the purpose of collecting it? So every metric is helping you to understand something. If you don't get the actual response or answer of what is the actual value of this, then question yourself.
Bart Farrell: And I don't think it's wild or crazy to be saying that As much as we can get deep into the weeds of the technology, going back to the basic fact of, or the question of, how does this advance business goals? How is this connected to the objectives of, at the end of the day, these are all organizations connected to some kind of business, and there needs to be some kind of objective linked to it, not just technology for technology's sake, not just collecting data for the sake of collecting data. All these tools have lots of shiny bells and whistles, but if they're not genuinely providing business value, then why are they there considering the cost that can go along with them? So I think it's never too often that we reiterate that so people don't lose sight of where they're going.
Salman Iqbal: That's fair enough. So Henrik, basically what you're saying is, yes, people collect lots of data. And if you collect a lot of data, it costs a lot of money. That's sometimes that can be a bit of a problem. What you're saying is when the science of collecting the right amount of data is to ask yourself a question. what value does this data or this metric or this event bring? Is that right? Is that fair? You have to ask that question over and over again.
Henrik Rexed: Completely. And then I had a case recently where people switched it from NGINX to Envoy. Fair enough.
Bart Farrell: Shots fired. Controversial.
Henrik Rexed: And then they just build the same dashboard but they're using Envoy. And then they don't ask the question how Envoy has actually produced the metrics. compared to how NGINX was producing metrics. And as a consequence, they were complaining because they were receiving, I think it was like, five terabytes per second of metrics per second. And then they were saying, oh, it's exploding our license and everything. And then I say, but your metric is produced at every single hit. It goes to the proxy. So the granularity of your metric is just insane. And the cardinality of the metric is insane. I think if I have a recommendation is if you do observability, you should start from dev. It's like a software. You validate things. And there's a component in OpenTelemetry called Weaver. So detecting things that could be expensive or things that you have to deal with a CI/CD pipeline, you could achieve this. And then discovering this in production where you will have 100 times more load and then 100 times more metrics could be kind of a pressure because then cost is involved and people will put you pressure, why this metric and what do you do? And then here you try to make a decision. So I think doing validations, understanding, and then moving step by step around the different stage, so dev, QA, whatever, I think it's going to be very important.
Salman Iqbal: That's fair. that's fair enough. That's great. So before we end and before we wrap up, we would like to look into the future. because everybody wants to know what's going to happen in the future. So let's have a look at this. So if the future isn't just thought, let's have more dashboards. Let me ask you a question. If you could redesign observability, Henrik, from scratch today, what would you build differently? What does the future of observability look like?
Henrik Rexed: So first of all, I think I always question why do we need dashboards? I mean, I think it's a human tendency. It's like a car. The modern cars, they have tablets, but they have less things that we had in the past. And I think sometimes we create dashboards just because we're sick of having a dashboard. But I think the data, from the moment it's stored in an efficient backend layer, it's queryable. So then you don't need a dashboard, you can query. If you have the right query and the right automation, you can make decisions. I think in general, I compare observability to a self-driving car. A car driving by itself, it has tons of sensors collecting tons of different data, and based on the data, it's taking decisions to react. And I think in the future, we should collect more things, contextualize more things, and you're not going to have a self-production driving car. No, we will still have humans, but having at least the data connected so then agents can take the right decisions without human contact. Because I think now, With the recent booming of AI agents, people tended to give to the agents tons of traces, tons of metrics, and then the agents are overloaded. And then you're shifting the costs to now token consumption. And then we say, oh, how can we make the agent more efficient? And I think in general, agents are great, but you can fine tune them. And then without the data, This is what you can see with the agent evals or other projects. It helps you to identify wrong or inefficient agents due to too much MCPs or bad skills or whatever. And again, if you don't have the data, how can you make the decision? So you still need observability. It will always be there. We probably need some dashboards to know where we're heading to, but I think mainly we don't need that. We just query and structure the data so the agent can get the proper information they need to get the right answers.
Salman Iqbal: And then just the right level of context for the agents, right? Because as you mentioned the example before, Envoy and NGINX, the five terabytes a second of traces are not going to help the agent at all. Is that correct? Just having the right level of information fed to the agents. Is that correct?
Henrik Rexed: Yes. And also, it's out of the discussion for today, but I think in general, we like JSON because we can easily do query on that, but agent doesn't care. They will perceive it as a standard string. So sometimes, do I need JSON? Maybe a compact markdown where the agent will be smarter. Also, retrieving data from a smaller time frame, so then you have less data, that will help the agent to be more efficient. Instead of sending him one terabyte of data that you have to process, this is where inefficiency comes in.
Salman Iqbal: Do you think there's some standards that are emerging in this area, like how to feed this information to the agents and how do they better utilize the tokens? Or is this a space that we should watch? Maybe check out your YouTube channel for this information in the near future on Is It Observable.
Henrik Rexed: I think it really depends on the vendor because in Dynatrace, we have a query language. Prometheus has its own query language. Everyone is doing query language differently. So I think if the agent has the skills to the proper backend that you're using, knows how to query things, knows how to utilize the things, then it's going to be smarter. But I think there's no standardized way to build skills for everything. Like I said, the way I'm going to retrieve data will be different from one backend to the other. So they still need to have specific skills, specific toolings for the agents that will match where you're storing the actual data.
Salman Iqbal: All right, excellent. This is great. Thank you so much, Henrik, for the discussion. So we started this discussion today with one claim. You don't have an observability problem, you have a decision-making problem. Henrik and Bart, after everything we've discussed. Do you think the thesis still stands or do you think we should rewrite it?
Henrik Rexed: I think we should rewrite it, to be honest. I don't know, Bart, what do you think?
Bart Farrell: Some nuance I think would be helpful as it came up in the conversation, particularly your comments around context. I think there's a lot to be said there, but how would you phrase it?
Henrik Rexed: I would phrase it, do I have the right context to let me understand my systems and to let an agent understand my system? So for example, I know that we're supposed to conclude, but this notion of topology. So in most of the system, we create, we redesign the architecture and the dependency of the components through the pod, from the service, from whatever. So then we know which service runs on that component, on that pod, or another infrastructure. This is by default in the schema of OpenTelemetry. It doesn't exist. It will come. There's an Entity SIG. If this has been standardized, then we're shipping data, and that data is already attached to a topology. And once we are there with this information, then context will be easier. But for now, be patient, but know your app.
Salman Iqbal: Be patient, but know your app. So to conclude then, Henrik says that you don't have an observability problem, you have a context issue. So make sure that you have the right context, just so you can make the right decision. And nowadays, it's probably, it's always been important, right? Because the context was always an issue. But even now, especially the context is more important as well. So appreciate that. Before we end, Henrik, let's have a last thought. If you had to leave platform engineers, DevOps people, with one idea about observability, what would it be?
Henrik Rexed: I know and I understand.
Salman Iqbal: And I conquer.
Henrik Rexed: Should we put that at the end? I think therefore I am.
Bart Farrell: Henrik, in terms of what's next for you, we know that you're going on a well-deserved holiday. You're going on vacation. After that, what can we expect from you?
Henrik Rexed: So I have plenty of content that I've been working on that will release in August-September. And then I have a couple of conferences, so Open Source Summit, Observability Summit, KubeCon of course. And then the conference season starts, so I will be very busy with the conference season. And in December, on the 10th of December, if you've never been to South of France, there is Aix-en-Provence, KCD Provence. So we are organizing the first KCD Provence in Aix-en-Provence on the 10th of December. So the CFP is still open until the 15th of September. So drop your CFP if you want to talk and be a bit there. And then we have the early bird tickets open that is until the mid-September. So if you want to purchase the ticket for a decent price, I would recommend to do it now. And we will have a lot of announcements about speakers that has been already selected and keynote speakers. So stay tuned on KCD Provence.
Bart Farrell: Very good to know that. Very exciting that you spun that up and things are moving in a very positive direction there. Lots of momentum. That being said, Henrik, thank you so much for joining. Salman, anything you'd like to add to finish?
Salman Iqbal: No, thank you all so much for joining for the show. And definitely check out Henrik's Is It Observable YouTube channel because, as you said, observability is here to stay. So, thank you so much, Henrik. And thank you so much, Bart. Hope you all enjoyed the first episode. We have plenty more to come.
Henrik Rexed: KubeSelect. See you.
