Brian Teller: In this episode of Kube Signals, I spoke with Mauricio Salatino about what changes when platform teams move from running traditional cloud-native applications to supporting agentic systems. We get into why non-deterministic agents create a very different observability and governance problem, and why platform teams need to understand how agents behave over time before they can really decide how much to trust them. Mauricio also walked through a pizza ordering application he built to explore this. What sounds like a simple agentic app quickly turned into a distributed system with multiple agents, state, coordination, and a whole lot of traces to make sense of. We also talk about agents running across Kubernetes clusters, CI pipelines, and developer laptops, and why the platform itself may eventually need to become the learning layer that feeds better context back into those agents. Here's my conversation with Mauricio. So for years, we've become pretty good at running cloud native applications. We understand microservices, Kubernetes, observability, and scaling. Your talk argued that AI infrastructure isn't actually the hardest problem anymore. The harder problem is what developers are building on top of that. Can you explain a little bit what you mean by that?
Mauricio (Salaboy) Salatino: 100%. Nowadays, when you're building platforms for running these kinds of applications, there are two different discussions. One is the infrastructure discussion: how do we run these things? These things can be normal applications plus LLMs. And that brings new challenges in how we route traffic. And the Kubernetes community knows how to deal with these topics. So we are making a lot of progress in that space. The problem that I'm seeing often is that the kind of applications that we are trying to build and the frameworks that we are using to build these applications are massively different from the kind of workloads that we were used to build. Not in the way that they will run, because we can probably package agents as containers and provide a runtime for that. But the behavior of these agents is completely different from what we did before. My gut feeling is that we will need to adapt our platforms to mitigate and to control these behaviors. When you talk about guardrails and agents doing things that they shouldn't be doing, now it's part of the responsibility of the platform to deal with some of that.
Brian Teller: How do those guardrails differ from the guardrails that we would already have in place, for least privilege and dealing with noisy neighbors? What's different now?
Mauricio (Salaboy) Salatino: The main part is dealing with non-determinism. You have agents that can do things that you didn't expect. You need to make sure that there is something in the platform that keeps track of how agents previously executed something. You can compare statistically to see if that's right or wrong. The presentation covered building trust networks. How much can you trust an agent to do something? After you build a platform that keeps everything secure, provides the agents with the right tools, services, and context, how do you evaluate how much you trust that agent to perform something? The only way is by collecting data about how the agents performed in the past. Our Kubernetes-based platforms have observability and keep data about what's going on for troubleshooting and alerting, but not at the scale of tracking everything an agent does to figure out how much you trust it. Our applications were deterministic before, and these agents are not.
Brian Teller: Since it's non-deterministic, how do we build a standard approach for all of the platform engineering teams, or does it have to be different for every team?
Mauricio (Salaboy) Salatino: No. A lot of new tools are popping up to solve different dimensions of these challenges that we are facing. One of the points there is that agents will run everywhere. They will not only run in our Kubernetes clusters, but they will also run in our CI pipelines, and they will also run in our developers' laptops. Because they are running all over the place in different runtimes, it's very hard for companies nowadays to understand what's going on. You need a way of understanding where your agents are, which context these agents are using, and you need that as a platform capability. Just as we had Backstage or a platform portal and our observability dashboards, we need a place where we can go and see what our agents are doing, where they are running, and how much they are spending. That's becoming more important now. We don't have the tools to do that in a unified way. Every company will have its own solution. Then we will build something that can be shared and improved by several companies. We're going through that phase right now.
Brian Teller: If someone missed your talk, what's the one thing that you'd hope that the platform engineers would leave understanding?
Mauricio (Salaboy) Salatino: For me, the main takeaway from the presentation was that we are solving a new set of challenges. We already have all the tools. We have the cloud-native ecosystem providing a shared vocabulary that we can all use. When we talk about GitOps, observability, security, we know what we mean and the tools that are going to solve these problems. But because we are pivoting to solving a set of new challenges, there are new tools popping up and we need to keep an eye on that. It's important for platform engineers to see what other companies are building and keep an eye on the standards and the standard formats or conventions that are popping up to solve these challenges. At Dash0, in the OpenTelemetry space, we have semantic conventions for generative AI. When you think about monitoring agents that are running in a lot of different places, having a convention that allows us to understand the agent name, the model that it's using, and the amount of tokens spent allows us to get information and correlate information to understand what's going on. We can rely on tools that we already know to solve new challenges. New tools will pop up, but part of platform engineering is adding new tools and capabilities to help other teams to move forward.
Brian Teller: You make the distinction between infrastructure problems and application problems. Why is that important?
Mauricio (Salaboy) Salatino: That's important because application developers think in completely different terms about the problem space that they are trying to solve. Application developers do not think about where that LLM runs. Am I consuming a service or is it something that is running in my company infrastructure? Running an LLM in your company infrastructure will take you quite a lot of time and resources to get that right. On the other hand, developers think about how to solve this business problem with this set of frameworks that will allow them to create these agents to be more autonomous. The whole point of creating agentic applications is to be able to create these solutions that can perform actions on our behalf without a lot of supervision. We are not there yet. We have the frameworks, we have the paradigm on how to build that, but the tools are not there. The problems that developers are facing are completely different from the ones that platform teams are dealing with: scaling LLMs in their own infrastructure.
Brian Teller: Do you think agentic applications are something new or are they a different kind of distributed system that we've always been running, but now think about it differently?
Mauricio (Salaboy) Salatino: No, it's a mix. In the presentation, I call them distributed multi-agent systems. The distributed systems part is well known in the Kubernetes world. The problem is that multi-agent systems are not new. I have a book somewhere there about multi-agent systems from the 90s. There's a lot of research about how to build these systems, but we are not using any of that now because we like to reinvent the wheel. My feeling is that in the next year or so, we will return to that research on how agents trust each other, share information, and share intent. That's why we do not have software-development tools and frameworks to implement them as they should. Every company will need to figure it out. Building these systems for customers can go wrong quickly. That's why on the platform side, we should build the guardrails.
Brian Teller: In your talk, you had built a pizza ordering application to explore all this. Why a pizza shop?
Mauricio (Salaboy) Salatino: The whole idea of building a pizza shop instead of building an SRE agent, for example, or something that's closer to the infrastructure, was to show that these agents can be customer-facing, and if they mess up taking your pizza order, you will not get your pizza, you will lose your money, and the company will go bankrupt. Also, I wanted to build small distributed applications to show multiple agents talking to each other, with each agent having its own model to reason about things. At the same time, not only the agents need to communicate with each other, but they need to communicate with other existing systems, with other APIs. That created a simple idea: agents taking pizza orders and cooking the pizza and delivering the pizza. The application became complicated very fast. The demo should be simple. It's probably 15 containers and a bunch of things installed in Kubernetes. When you run one pizza order, you get 200 traces to figure out what's going on. Understanding the interactions is complicated. The funny thing is nothing worked out of the box. I needed to patch some of these frameworks to troubleshoot when things go wrong. By creating a simple application that is customer-facing, I could find all the gaps and explain to people how difficult it's going to be to build these solutions.
Brian Teller: When you were building that, what looked simple? You mentioned that it was more complicated than it initially looked. Can you dive into that?
Mauricio (Salaboy) Salatino: From a developer's perspective, there are three things I highlight in my presentation. Skills drive me a little crazy because they provide context but also bring some mechanisms that I'm not happy about and that are difficult to observe and to secure. I also looked into MCP: how easy it is to troubleshoot and observe, and how chatty it is, and how much configuration you need to understand what's going on. Finally, the other protocol that joined the Agentic Foundation is A2A, which enables two agents to communicate with each other. Claude Code and Codex are completely different from business agents. I built a pizza application because I wanted to make sure that people do not confuse that with coding agents. Those are completely different tools.
Brian Teller: Once you got to the point where the pizza agents work, what suddenly became the platform team's problem?
Mauricio (Salaboy) Salatino: Observability.
Brian Teller: Observability.
Mauricio (Salaboy) Salatino: Observability was the first thing because everything was broken. It becomes important to provide mechanisms across these frameworks. This is a problem on the developer side. Depending on the framework that they choose, the platform can provide those capabilities instead of pushing the developers to solve these problems. In this case, observability was broken in multiple frameworks. Understanding how the pizza order goes through the entire system was difficult to achieve. It took me a while, even though I expected it to work out of the box.
Brian Teller: How did you deal with state management?
Mauricio (Salaboy) Salatino: State management is always a problem. One issue is the state of the agent itself. If you have multiple agents, you have the states of each agent and you have the orchestration across agents. How do you manage the state of the order? If you have three or four agents delivering an order, how do you determine which agent does what and, when that agent finishes, move the order to the next agent? I had a couple of versions of the applications using different tools to solve the state. At some point it became too complicated, so I removed the state management to keep a simple version of the application working. When you add state management, you add more storage, distributed transaction patterns or saga patterns. You need an orchestrator.
Brian Teller: What about governance between the agents?
Mauricio (Salaboy) Salatino: That's the next issue. This was for a presentation, but if you're running at scale, responsibility for each of these agents becomes important. That goes back to skills. People tend to think that you create an agent and you give it the right context in the shape of skills, and the agent will perform its work. The question is: how do you evolve that agent moving forward? How do you version the skill? How do you track which skill was running when this failed? Each framework has an opinion. The problem is that platform teams do not know about these frameworks. They do not know about the gaps in these frameworks. They will get workloads that run agents that they don't fully understand. We're going through that period of multiple things popping up. Platform teams will need to focus on core capabilities: running LLMs, versioning skills, and providing skills registries. I've seen companies trying to go into that space. But there are many problems we haven't solved yet.
Brian Teller: Looking back at what you did with this pizza app in your presentation, if someone builds their first agentic app this month, what mistakes are they almost guaranteed to make? What assumptions have you already seen?
Mauricio (Salaboy) Salatino: The main mistake I see repeatedly comes from thinking about microservices. Because we come from that space, we think about one microservice, one container. When you pick agentic frameworks nowadays, some of those frameworks will push you to create multiple agents inside the same container. That's where scaling the application starts to resemble scaling a monolith: it's a single runtime for multiple agents, and they will battle for resources. When you run that in production, you start hitting big issues.
Brian Teller: Was there a failure that taught you the most while building this pizza application?
Mauricio (Salaboy) Salatino: State management was one of the first failures: trying to ask an LLM to follow steps, step by step. You're creating structures to say, first you cook the pizza, then you deliver it, then you charge. That's the wrong approach. It works sometimes. Depending on the model that you're using, it will change. If you are using a public service, it will upgrade the model and that will change. You never know. Testing that is bad. It takes time and you're never sure that it's going to work. State management or coordinating multiple steps is difficult. My lesson was: build agents that are scoped to one task, not multiple steps. You can have multiple agents doing concrete tasks, and they can send the tasks to each other.
Brian Teller: Observability. It all comes back to that. Where do we start with observability? Let's say we have Grafana, Prometheus endpoints, and traces. What's different with this agentic platform?
Mauricio (Salaboy) Salatino: The main difference with agents is understanding how the agent reasons. When you're creating a trace, one of the attributes of the trace is the prompt that was sent to the LLM. Another attribute will be the completion or the reply from the LLM. These tools haven't had a way to represent this kind of conversation. Auditors or whoever is responsible for monitoring these solutions can check one of these sessions to see what happened. We have other solutions, but we need tools that are shared across companies based on open standards. Going back to semantic conventions, that's what that project that sits inside OpenTelemetry is trying to figure out. What are the properties that are specific to agentic applications that we need to track? But on the UI side there is still a lot of work to do to have the right tools for someone to say, this failed, I can troubleshoot and find what happened there. In my demo, I show what an entire end-to-end pizza order looks like. You can see which services were called, which parameters were used to get from the inventory to cook the right pizza. The system becomes more complex as it grows. It goes completely out of hand. It's a lot of new information that you need to process.
Brian Teller: If they're building an agentic platform, what do non-deterministic agents change about observability?
Mauricio (Salaboy) Salatino: The amount of data that they will need to record to verify that these agents did what they were supposed to do. In my presentation, one point is that you need to record all these interactions to understand how much you trust that agent. If you have 10,000 pizza orders and the agent messes up in only 100, is that good enough or not? You need to build that database of executions to understand if the agent is doing what it's supposed to be doing.
Brian Teller: Is there a point where we get too much context and it muddies the observability too? How do we reason about what we're saving versus what we're not or how we're indexing that data?
Mauricio (Salaboy) Salatino: The first step is getting the data. For people listening, I recommend recording what your agents are doing using OpenTelemetry. You store that in a standard format that can be sent to different backends. You need to correlate that information and compress it so you can store the things that you need to reason in the future about how the platform is working and how your agents are performing. The last part is how you get all the information you learned and push that back to the agent so they have the right context to do better in the future. One point in the presentation is that LLMs do not learn after they are trained. They will keep doing the same task in the same way. Your platform is the one that needs to learn now so it can provide these learnings back to the LLM as context. We do not have the tools for that, at least none that I'm aware of. There might be companies building this. If the tools are not there, each company will try to figure it out on their own.
Brian Teller: Moving to trust: we have a human in the loop. When do we trust an agent enough to stop supervising every action and letting it autonomously do its job?
Mauricio (Salaboy) Salatino: That's the main question we're trying to answer with coding agents. We're getting too many pull requests, but we have tools to review them. What does that mean? If the tool says, yes, merge, do you trust it or not? What happens when you merge something that you didn't want to? That's what's coming. One point from the presentation is that you need to record not only the data about when your agents are being executed, but also when your pipelines are running, and when a customer issue gets reported, because you need to use this data to correlate things. Imagine that you have an agent creating a pull request and another agent that reviews that pull request. You need the data about how many pull requests you merge that were reviewed by that agent to understand how much you trust that agent. This creates exponential growth in the data that we need to capture and reason about. It's something that we have done in the past because data always keeps growing. But the ways that we compress this data and use this data are different from what most platforms do. This feels more like a data platform than a normal workload platform.
Brian Teller: What should platform teams build over the next year? Given there are no clearly defined golden paths yet, how do we use these tools safely and follow the same trajectory as the rest of the industry? How do we stay within those guardrails?
Mauricio (Salaboy) Salatino: I always suggest starting by understanding what's going on and collecting data. In the presentation, I show a dashboard that lets me see all the agents that are distributed in different places with different frameworks and different tools: my Claude Code session on my laptop, my CI agent that is reviewing a pull request, and my SRE agent that is troubleshooting production environments. As a platform team, I want a dashboard where I can see all these agents and understand how much time they spend solving things and how much money they spend on tokens. Which models are they using? Are they effective or not? In my Platform Engineering book, I left observability or maybe metrics towards the end, in the last chapter, and I keep regretting that because if I wrote it again, I would put it first. If you're starting a platform, measure how your deployment pipelines are performing today. Then you know how they will perform tomorrow. Most companies are building and changing agents without knowing if they work better.
Brian Teller: Any closing thoughts for the audience on building agents, your work, or where you think the industry is going?
Mauricio (Salaboy) Salatino: I recommend looking into multi-agent theory that was written in the 90s. That's the nerd in me. I want to use some of that because I remember it being useful and aligned with what we are trying to build today. On the other hand, don't use coding agents to generate pull requests that nobody will merge or review. It doesn't make much sense. I encourage people to be more mindful about that. If you're trying to build your own distributed multi-agent application, I recommend sticking with one framework and finding the gaps and the topics where the platform team can help you. As a developer with these frameworks, you tend to think that you are going to own the entire thing. You will write the software and run it in the same way you run it on your laptop. But to scale these applications, that needs to change.
Brian Teller: For sure. Thank you so much for coming on. I really appreciate your time.
Mauricio (Salaboy) Salatino: Fantastic.
