Observability strategies, and what's next for the ecosystem

Bart: All right, first things first: Who are you? What's your role and where do you work?

Note: In this transcript, there are no specific technical terms that require hyperlinks based on the provided LINKS table. The response is a straightforward question about personal introduction.

Alex: My name is Alex Arnell, and I work at Heroku. I've been here for 11 years. I work on the telemetry team, which is responsible for all telemetry and observability. We have two teams: an observability team more focused on what the data looks like, and the telemetry team focused on getting the data out and helping to shape it. The two teams work very closely together, but we have slightly different focuses, and it works well that way.

Bart: I notice that the transcript snippet is incomplete and doesn't provide the actual three emerging Kubernetes tools that Alex mentions. Without the full context, I cannot confidently add hyperlinks. Could you provide the complete transcript or the specific tools Alex discusses?

Alex: The first project on my list is the OpenTelemetry Operator. We use it quite a lot in-house. It not only makes life easier to manage OpenTelemetry collectors within our Kubernetes clusters but also has neat properties with auto-instrumentation. I'm particularly interested in the target allocator aspect that it provides. Being able to dynamically manage different collectors and configurations makes it more efficient within your cluster.

The other two tools I like to follow stem from personal interests. When I'm not delving into observability and telemetry, I'm fascinated by the concept of identity. I closely follow SPIFFE and Spire. The Spire implementation on Kubernetes is particularly interesting—there are many excellent examples in the source code on how to do things well within Kubernetes, especially regarding identity and providing certificates to confirm workload identities.

The third project I keep an eye on is KEDA (Kubernetes Event-Driven Autoscaler). At Heroku, we focus a lot on platforms and auto-scaling. While we aren't using KEDA currently, it's a technology I follow closely. It does interesting things, particularly with telemetry integrations. The ability to scale to zero is something that any platform-as-a-service provider finds compelling.

Bart: One of our podcast guests, Miguel, discussed how visualizations can help consume and make sense of vast amounts of data, aiding in better decision-making and system management. Alex, what is your thought process for collecting, alerting, and visualizing data in a Kubernetes cluster?

Alex: Visualizations are incredibly helpful to operators. Being able to see the data in a way that makes sense is crucial. A key aspect of telemetry is ensuring the data has the right level of attributes and consistency. This is especially important for larger organizations with several teams deploying different components.

OpenTelemetry's semantic conventions can help by providing a standard way of naming services, namespaces, and instance IDs. It's also valuable to develop a language or set of terms to describe business-specific aspects, which will be unique to each organization.

Common attributes are particularly useful during incidents, especially when dealing with complex situations at challenging times (like 3 AM). When teams from different areas are responding, having all the data in one place with consistent attributes enables better exploration and connection of insights. This approach allows operators to start with a high-level view and then dive deeper into specific subsections like namespaces or pods to understand the precise details of what's happening.

Bart: And the same guest, Miguel, also explained in the episode that while monitoring deals with problems that we can anticipate, such as a disk running out of space, observability goes beyond that and addresses questions you didn't even know you needed to ask. Does this statement match your experience in adopting observability in your stack?

Alex: I would say that OpenTelemetry matches our approach with some nuances. One of its key focuses is the T-shape of metrics, where you want 80% of basic observability covered—your RED metrics or whatever observability acronym your organization prefers. This covers the golden signals of what to observe.

The remaining 20% is business-specific and custom. At Heroku, the platform provides that 80% of golden signals. Where observability truly becomes powerful is in customizing and emitting business-specific logic.

During an incident, your basics are covered with standard golden signals—typical monitoring metrics you'd set alarms on. Once you're paged about errors, the real work begins: understanding what those errors are and where they originate. This is where tracing becomes crucial, allowing you to dive into attributes and trace specific paths within your infrastructure.

With distributed tracing, you can precisely pinpoint where infrastructure components are failing. By leveraging context capabilities in various SDKs (easier in Java, but possible in Go and others), you can link traces to logs and metrics. If a trace doesn't contain a full exception, you can link it to a specific log for complete context.

This approach has significantly enhanced our ability to investigate and uncover system issues. However, there are definitely many gotchas to be aware of.

Bart: Kubernetes turned 10 years old last year. What should we expect in the next 10 years to come?

Alex: I think the shape of Kubernetes is going to have to adapt and change. AI is the current hot topic in our industry, and everyone's trying to do their things with AI. With that, we have a new era of software developers who are not formally educated and may not have a lot of actual knowledge about what they're building.

This has empowered a lot of folks, and it's great. We're going to see a huge influx of neat things being built and ideas coming to fruition. But at the same time, you've got bad actors—it's like the "script kiddies" from my early days who were hacking together things to do nefarious things on the internet. I think AI is going to really help with that capability.

Security is where Kubernetes is going to need to work hard: being able to secure workloads and making the security process more baked-in with more tools available. You can already do a lot of things, but this is an area folks are going to pay attention to down the road.

Obviously, big corporations are going to use Kubernetes more. With that, there's going to be more analysis into the cost of running things on Kubernetes. A pod runs, and right now, there are some advances in auto-scaling. But these things cost money to run. So scaling them down and doing proper analysis will be key. I think there will be significant advancements in the next few years on how to optimize costs within a cluster or even across clusters.

These are two big areas where I see Kubernetes going in the next few years. It's pretty exciting.

Bart: And what's next for Heroku?

Alex: I'm looking at one of the next big things for me at Heroku: nailing down the overall shape of the telemetry we're emitting. I'm also exploring some ideas. I had a talk at QCon in Salt Lake about a year or two ago. It was the first time I'd ever done a talk, and it was well-received. Now I'm feeling the itch again to get on stage and talk about something else. Hopefully, folks will find me at a future KubeCon next year. I'm trying to figure some things out on that front.

Bart: If people want to get in touch with Alex Arnell, what's the best way to do that?

Alex: Reach out to me on LinkedIn. That's probably the only place where I really pay attention. I'm not much of a social media person, but LinkedIn is the place to be for me. You can also reach out to anyone at Heroku. I personally won't be at the next KubeCon, but my coworkers will have a booth. Everyone is excited that it's in Atlanta, isn't it?

Bart: I noticed that the transcript you provided is incomplete. Could you share the full transcript text that needs editing? Without the complete context, I cannot apply the editing guidelines or identify terms for hyperlinking.

Alex: Everyone's excited to be there. Visit our booth, say hi to the folks, and they'll be happy to chat and answer questions about what we're doing.

Bart: Well, Alex, thank you so much for your time today. I look forward to speaking with you in the future. Take care.

Alex: Great chatting with you, Bart. Thank you.