Unpacking observability, ditching Prometheus

Bart: In this episode of KubeFM, I got a chance to speak to Adriana, who's a senior developer advocate at ServiceNow, formerly known as LightStep. But beyond that, she's got some interesting and somewhat controversial takes when it comes to the topic of Prometheus, when it's appropriate to use it, and when perhaps it isn't. She's been working with OpenTelemetry for quite some time, some great stories about how she's helped people level up when teams have faced technical challenges. And on top of challenges, it would make things very, very interesting. I've been recording podcasts for quite some time and never have I had the chance to do a podcast with a mother-daughter combo, like the amazing team that Adriana makes with her daughter, Hannah, who helps her out on their podcast called Geeking Out. Pretty sure you're going to enjoy this episode. I know I certainly did. Let's get into it. All right, so we are here in a very, very special episode of KubeFM. Welcome to KubeFM podcast. Adriana, it's very nice to have you with us today.

Adriana: Super nice to be here.

Bart: Great. That being said, we will have a lot of surprises in this episode, but we'll leave them for a little bit further down the road. We will be having a special guest with us, a very special guest, by far one of the most unique people I've ever met who has participated in CNCF. We'll get to her later on. But just right off the bat, Adriana, if you had to take three Kubernetes tools to install in a brand new cluster, which ones would they be and why?

Adriana: Ooh, it's like the deserted island version of Kubernetes. Let's see, I think Argo CD would probably be one of them because I mean, like deploying to deploy applications to Kubernetes cluster before the age of like GitOps tools, like why? So it just makes life so much easier. So that's number one. Number two would probably be the OpenTelemetry operator near and dear to my heart, because it's part of like the OpenTelemetry is part of the CNCF ecosystem, just like Argo is. And it's a great little tool because it manages the deployment of the OpenTelemetry Collector for collecting instrumentation data from like your apps and infrastructure. It allows you to automatically instrument your applications so that if you're getting started with application instrumentation, you don't know where to get started. You can, you don't have to touch your code, just update or add a CRD away. You go not, not quite that simple, but it does make life easier. Um, yeah. So a OTEL collector and let's see third, um, The third Kubernetes tool. I'm kind of brain farting right now.

Bart: Two's a good start. I'm sure we'll find a third. We'll get to a third along the way.

Adriana: And I think this is also- It popped into my head like all of a sudden. Aha!

Bart: Two out of three ain't bad. I think it's also, this is a perfect segue into introducing the third person that's in the podcast with us right now. who happens to be very near and dear to you, Adriana, but we'll let her introduce herself. Hannah, very nice to have you with us at KubeFM. Could you introduce yourself? Tell us who you are.

Hannah: My name's Hannah. This is my first time on someone else's podcast, so I don't really know what to do.

Bart: Tell us who you are.

Hannah: Well, I am a high school student. I am... I don't know a lot about tech, so that whole mumbo-jumbo she was saying just sounded like gibberish to me. And, yeah, I don't know how I feel about tech yet. I used to not like it, now it's warming up to me. After taking a computer science class, you know, it's warming up a bit. Um, but it's definitely not my thing.

Bart: That's good, and it's good that you're here with us. Also have to add just a small tidbit that Hannah is not just a random teenager who we invited to the podcast. She happens to be Adriana's daughter and it's just worth mentioning. So anyway, we're an experimental podcast and this is no exception to that rule. But with that in mind, we will be getting in touch with you, Hannah, later on, and your feedback is very valuable and it's a way to make sure that your mom, Adriana, The truth and nothing but the truth. All right. So we'll keep that going. Good. So that being said, all right, a little bit of background on you Adriana. How did you get into the whole Cloud Native Ecosystem? Tell us about where you're working right now. What's your background?

Adriana: Yeah, so I work at LightStep, which actually, I guess, we got renamed to ServiceNow Cloud Observability, so formerly LightStep. My foray into the cloud world, I guess, a little bit of a baby step when I was working at the Bank of Montreal here in Toronto, Canada. they like, there was like a big, like, there was a DevOps department, if you will, that was created to kind of, you know, thou shalt do DevOps throughout this organization. And so our, our team, I was running a team that basically helped bring DevOps awareness to the organization, DevOps practices, and just getting people jazzed about DevOps. And of course, one of the tools in our tool chain was OpenShift, which was like my first into the world of Kubernetes. I like to call it sheltered Kubernetes because there's like so much, I mean, it's still complicated but there's still so much abstraction on top of Kubernetes that there's like certain things that you take for granted when you're using OpenShift that when you're using like plain old Kubernetes, you're like, oh, well then that's what's happening behind the scenes. So I guess that was my first introduction into the cloud native world. Mind you, we weren't, on the cloud at the time it was on-prem. So my real like cloud native experience was after that when I moved on to a release engineering team at a company called Ceridian. And they are predominantly a Microsoft shop. So that was like my introduction into Azure. And honestly, prior to that, I thought this whole idea of like cloud native was... Oh my God, this is this terrifying thing. Everyone's doing it. It sounds so complex. And then I just realized, oh, it's just infrastructure in like someone else's data center. And there's like APIs to provision stuff. Cool. That's awesome. So yeah, that's how I got in.

Bart: Very good. And if you could go back in time, you know, having gone through that process first with OpenShift then getting introduced to Azure, what were things that you would tell yourself at that time in terms of things that would be good to keep in mind regarding resources, learning, leveling up, upskilling?

Adriana: I would say, first of all, don't be so scared of it. Cause initially when people were talking about cloud native, I thought, oh my God, this is terrifying concept. There's no way I could possibly learn this, which is ridiculous. I come from a software engineering background. I've had to retool so many times throughout my career. So yeah, just tell myself to not panic because I think that's probably the best thing the most important thing. And just to like, ask lots of questions, be curious, I think it's really important to constantly be curious. Don't be don't be okay with like, oh, it works, but like, why? Right? I think that's, that's a big trap that we can fall into in our industry, especially like when you're working on some code and you it's like hobbling together on band-aids and duct tape, you're like, oh, it's working. Yay, I won't touch it. But you kind of have to you will eventually. So just not being afraid to like, Try to understand why it is that it's working that way. Not being afraid to break your stuff too, right? Because I think it can be like so terrifying, you know, you got something working, and it just like hanging by a thread. You don't want to touch it anymore, but if you don't, how can you improve it?

Bart: Great points. Yeah. The only fear we have to have is fear itself, right? That it's okay if things break, it's part of the process. Now we know after this podcast that Hannah is going to be so inspired that she is definitely going to go ahead into a career in cloud native. So if you had to give her advice, if and when she chooses this as a career path, what advice would you give her?

Adriana: Um, probably don't be in the same room as me while I'm coding so that you're not turned off from it.

Hannah: She is so scary when she codes. It like, she swears a lot and you're like almost afraid to even move because like it might just make everything explode.

Adriana: It's not just me, I mean my husband's in software too.

Bart: With that in mind too, I think it's a good reminder that as much as these experiences are very technical, at the end of the day, they're being done by humans, and naturally they're going to be very emotional. And Hannah, you are an active witness of those ongoing emotional processes.

Hannah: Definitely.

Bart: Good.

Adriana: I would also like to add that, you know, like the other thing for getting into, like actually answering the question on getting into Cloud Native, which is like, you know, treat it as like, it's creative work, really. I know a lot of people don't look at the software industry as creative work or even the tech industry in general as creative work, but it really, really is. And for someone like Hannah, who is a very creative person, who might not be necessarily attracted to this type of work, because she's like, well, it's not painting or baking or like, whatever. But it is like it still involves a huge amount of creativity. And I think just don't be held back by the fact that it's like, you know, it's technical work. Just allow, allow. I think the more creative you are, the more successful you can be in this industry.

Bart: It's a very, very good point. And something that's often overlooked about just how much creativity goes into it in the same way that you find whether it's with architecture or art or music or baking. Spoiler alert, we will be talking about baking later. That's a hand to somebody to talk about. Also, you know, and you both of you are involved in other podcasts. Could you talk a little bit about that?

Adriana: Yeah, so Hannah and I have a podcast together called Geeking Out, which we started up started it up after my previous podcast On Call Me Maybe came to a close. And you know, like, so that one was associated with with my actual workplace. This one is not associated with with my workplace. So I saw it as a good opportunity of of it being rather than attached to like, It still works, but it's more attached to personal brand. And since Hannah actually has an affinity for editing videos and whatnot, we thought it'd be really fun to do this as like a little project where mostly I'll let Hannah actually talk about her involvement in the podcast.

Hannah: So, well, it started with me doing the graphic design for it. So I created the logo with the Cat on it, which I'm very proud of. And just like all the little banners for all the socials and the backgrounds for the, yeah, for the YouTube videos. And then I also do, editing for it. Mostly it's the subtitles. So whoever's doing, if like someone's doing the subtitles for this, I feel you. It's very difficult. But yeah, and then I like put it all together. And then I also help with like the social media of it. Like sometimes I do some posts, like I'm gonna try to make it a tradition to post on our stories, embarrassing pictures of her when I pause it. And she's like, So yeah, look for that on our accounts.

Adriana: Such a troll. Yeah, on our Instagram especially.

Bart: I think it's really healthy because it brings these things to life in a different way that if not can sometimes be difficult to grasp or can be left in that area that's too technical and overshadowing that part that really is creative and human. So Hannah, keep up the great work. That's fantastic. That's great. And Hannah, since you're listening to a lot of podcasts, do you have any advice for people when it comes to public speaking, such as myself in this particular role?

Hannah: Well, I think definitely if you're on a podcast, like don't think about how it's gonna turn out, just think about the conversation that you're having with people. Because sometimes if you think, oh my God, so many people are gonna see this, that's when public speaking fears come in. But like, If you just think, well, I'm just having a chill conversation, it's going to be a lot easier. And just for in general public speaking, don't worry about what other people are going to think about it. Think about why I'm doing this and what I want to accomplish by saying this.

Bart: That's fantastic. I think that's beautifully said. And so often getting caught up in, you know, details here and there forgetting about overarching objectives, but what is that I'm trying to do and how this conversation is helping me get closer to that, rather than like you said, feeling that that embarrassment or that shame that's going to prevent you from doing the things that you want to do to help you achieve your objectives. That's a great point. This is really good. All right, cool. Now we're gonna shift it back over to you. Adriana. This topic of observability, what was your, you know, you mentioned a bit about your background. When did you decide, all right, this is really where I think I'm going to make my mark? What was that? What was that? How did that, how did that spark it?

Adriana: It was a little accidental, as all things are, I feel. Like, the only reason I even like heard about observability one time, initially, is because like, I happened to see a tweet from Charity Majors on observability. And so that was kind of like, oh, this thing looks cool. And I had, When I was at Ceridian doing the release engineering work, there was some talk about, like, I was doing a lot of work on Kubernetes and on Argo CD, and they were considering moving their workloads to Kubernetes. And so there's talk about observability. And I'm like, well, you know, we should, because they were going with the Microsoft stack. They were kind of married to that. I'm like, you know, we should look at other tools just for funsies, right? Just to see what else is out there. And that was like the extent of the conversation on observability there. And then the job I had after that, which was at Tucows, which you may remember Tucows from the olden days of the late 90s, early 2000s, as that place where you could download the freebie Windows software, which was it was the same Tucows, but different business model by the time I joined, because they were basically a domain wholesaler and they did some, they split off into another area that was like mobile phone, like some card provisioning. So same Tucows, different business. But anyway, so I found myself managing a couple of teams, one of which was an observability team. And it was one of those, oh shit moments where I'm like, I'm managing an observability team. I am providing the direction for this team within this organization. I better know what I'm talking about. And so I had to, you know, educate myself more than just like, you know, seeing a couple of tweets and whatnot and just really dig into like, hey, what is observability? Why do I care? Why is this important? And, and, and, and, and, and, and, and, and, and, and, and, and, and, and, And so, um, as part of that exploration, I do what I always do, which is when I discover things, I blog about them to try to distill them for myself, but also for others who are probably struggling with similar sorts of concepts. Um, so as I'm exploring, I'm blogging and I'm also like leading this team and, and, um, bringing them, uh, like my, my ultimate goal was like, it was an observability tools team. And I'm like, no, we're, we're doing real observability. Like let's focus on observability practices. So I'm like, paying a vendor for this stuff already, why are we managing tools internally? Like, let's focus on observability practices. Let's focus on making sure that we're really following what observability promises to deliver within this organization. So that's how I came to really like dig deep into observability. And then because of all the writing that I was doing around that as I'm exploring that and exploring OpenTelemetry, that was like my first foray into OpenTelemetry, which was like of a baby at the time. This was back in 2021 and Traces, which was the first major signal of OpenTelemetry that had become available, was not even generally available at the time. And I'm trying to tell this organization, trust me, guys, it's going to be big. Really, I swear. And lo and behold, OpenTelemetry now, I think it's got the second highest number of contributions in the CNCF. Only behind Kubernetes. And it's got the support of all the major observability vendors. So this is not some like fly by night operation. Like we've got, there are things, there are many wonderful things happening. And it ended up leading me to my current job where, you know, I was like blogging, doing my thing that I love to do. And then my former boss, but person who hired me into LightStep was like, hey, how would you like to do this for a job? I'm like, what? These things exist?

Bart: Well, luckily they do because it's a very necessary gift for the community to be able to receive in the sense that what we talk about a lot of the podcasts is about leveling up. And so through writing, through sharing that knowledge, a lot of times there are really, really intelligent people that can work on these technical concepts. When it comes to translating that to other audiences, bringing stakeholders in to help make decisions around how these technologies are moving forward, that's a completely different story. So I think it's, I think it's a really good thing I think that's something we can touch on a little bit later, because I liked how you mentioned the point of when you found out you were going to be leading a team, I can't run this leadership based on reading a couple of tweets. You know, I've got to dig deeper. And so in order to dig deeper, there need to be people creating resources. And that's, I think, falls in line very well with what you're doing nowadays. But in terms of when you're modeling observability, how do you do it? Do you trace apps, clusters, metrics? How do you go about it?

Adriana: So my philosophy on observability is basically looking at it from like a trace-first approach. And the reason why I say that is because the traces give you that overall picture of what's going on, right? When you're on a website and you're clicking add item to cart, the trace enables you to see exactly all the little things that happen from the moment you click. click that button to the moment a thing is added to your cart. So having that kind of visibility is key. Now, that's not to say that, you know, the logs and the metrics of the world aren't important. I think they are very important and they provide a set additional contextual information, but I don't think that they are the, I don't think they are the rulers of the landscape. They're more like the supporting actors.

Bart: Now, let's say hypothetically, someone has ELK installed on their stack and they're exposing PROM metrics, PROM Prometheus. Is that enough? Could this be called observability already? Or would you say no?

Adriana: I mean, some people might argue with me and say, well, you know, like, you got to start somewhere, which, okay, I'll give you that. But is it true observability? I don't think so. Like metrics alone aren't going to give you what's going on. Because I mean, like, and they're important, right? Like metrics are the building blocks for SLIs, which in turn are the building blocks for SLOs. And those are the things, or service level objectives. Those are the... are targets for reliability, right? And without those, and, you know, like, I think that's an important piece of any SREs toolkit, right? Because they tell us that there is something wrong. But they can only tell you so much, right? They can say, okay, yeah, this SLO was breached, because whatever, you know, whatever condition wasn't met, but then you have to kind of dig deeper and understand, like, what are the things that But what are the things that were happening that led to this? And a metric can only take you so far, right? Because it's aggregated data. And it just tells you about, you know, one aspect of your system, whereas the trace will tell you like, it stitches that picture. It's almost like the history, it's the history of your call, right?

Bart: Wow, good. And, We're getting to the point where Hannah's starting to yawn. So now we gotta get really controversial to bring her back into conversation.

Hannah: no, no, I'm sorry. I've had a long day at school. I'm just tired after school.

Bart: We cannot have another yawn. All right, we cannot have another yawn. We know that you have been dying all day to hear your mom talk about SLOs, SLIs. She didn't mention SLAs. So Hannah, can you tell us really quickly, what's an SLA?

Hannah: Simple line attribution.

Bart: That's what it is now. So we have a new definition. You heard it here for the first time. A simple line attribution. This is Hannah jumping in doing a very good take on how the industry sounds. Wow, that was fantastic. So that's definitely going in a highlight clip. Simple line attribution. Well done, Hannah. Great job thinking on your feet. I think with this in mind, though, it is a good point about... about where this might fall short. And so sometimes we don't want to fall into just the oohs and ahs of a buzzword, no matter where it's coming from or who's saying it. So let's get a clear look then, Adriana. If you have to build an observability stack, what's going to be in there for you to call it observability?

Adriana: Yeah, so for me, I... So I don't know that anyone is doing this right now, but in my ideal world, having a unified SaaS tool that can handle all three signals, the traces, the metrics, and the logs, so that not only are they capable of ingesting them, but also capable of showing the correlations because I don't think we necessarily have that right now, but that's not to say like the vendors are working towards that, which is awesome. And I think whoever manages to do that the best is definitely gonna have an advantage. So I think having a SaaS tool that takes care of that. That's not to say that you can't run your own, like, you know, there's open source tools that you can run on-prem and, you know, more power to you if you wanna do that. When I was doing that sort of thing at Tucows, I would rather just pay the experts to do the thing than to have a SaaS tool. let them take care of support and the reliability and resiliency of the tool, and I can just focus on the observability bits. And I think the other part is making sure that your application is instrumented. And it's not just instrumented, but instrumented correctly, right? And I'll add one more thing to that and say instrumented with OpenTelemetry, because it is becoming or is already the standard for instrumenting your application or infrastructure. Let's stick with the standards, because that way, if you do change to a different SaaS vendor or you decide, hey, I want to run the stack on-prem, you don't have to re-instrument all your code. Right? It's already instrumented with OpenTelemetry. Boom, no problem. We good. But just because you've instrumented your code doesn't mean that, you know, boom, done, you know, problem solved, observability in hand. It's like, you know, the old DevOps thing of like, I have a CI/CD pipeline with Jenkins. I have DevOps. No, you don't. You have a CI/CD pipeline with Jenkins. That's all you have. Similar sort of thing, right? Like, you know, You know, we had, I'm part of the OpenTelemetry end user working group, and we had a couple of weeks ago, Hazel Weakly come on and talk about some of her experiences where she was working at an organization where they had instrumented their code using OpenTelemetry, but they instrumented wrong. Like they were just like, it was almost instrumentation for the sake of instrumentation. And as a result, they weren't getting anything useful out of their instrumentation. So really. instrumenting things that are meaningful, that are going to actually tell you the things that you need to know, so that you can actually get the most out of observability, which is being able to answer the question, why is this happening?

Bart: I love that, and I love how the point that you mentioned about seeing some folks that are out there maybe doing instrumentation for the sake of instrumentation. We're about to be in KubeCon in Chicago, and sometimes walking through the booths, I feel like maybe you're answering a question that nobody asked. There are a lot of vendors in the observability space. There's been a lot of growth, like you said, after Kubernetes, it was extremely, the second most active project in the CNCF ecosystem, which is no joke in terms of contributors and the attention that's going onto it. For all the vendors that are out there, we know there are gonna be trade-offs, we know it's not one size fits all, there's no silver bullet. But in terms of approaching end users' concerns, what advice would you give to vendors that are out there?

Adriana: Listen to your end users. I mean, like, really, really, really listen. And I think, especially in the observability space, like, we have the end user working group, where we actually have, like, we run a few sessions every month where we have, like, our end users either doing a Q&A, they talk about the observability landscape in their organization, or they do, like, we have a style thing called OTEL in practice where they present on an observability. topic. And in these sessions, there are also opportunities for these end users to share like their experiences, feedback with OpenTelemetry. And these are such a great way to really like to get that feedback. And also like we have an end user survey for OpenTelemetry. Again, just to like just. being dialed into the feedback for end users. And especially like if you're an observability vendor that is saying, hey, we support OpenTelemetry. I think being dialed into the OpenTelemetry community is going to help a lot. And I think listening to these user stories, user experiences gets you to where you need to be really.

Bart: Really like that. And having been involved in different communities for quite a while, people use the word community left, right and center. Like it's, you know, something they can just give out for free, but I think anyone that really knows what it's about to be, what it's really about being in a community means showing up, paying it forward time and time again, asking questions, answering questions, really showing that you care about people and that it's empathy based. You know, it's interesting that we're talking about the importance of creativity, but I think we cannot. understate or overestimate the importance of empathy and all this and really dilate into what what's frustrating people and trying to help them and not just saying, Hey, I got this brilliant solution without, you know, listening to their concerns first. So I really like that. One thing that you didn't mention and Hannah, we're getting to another controversial point here. So you definitely got to pay attention. No yawning. So controversial point. That was actually the original point that sort of caught our attention and why we got in touch with you, Adriana, as you didn't mention Prometheus when you were talking about that observability stack. Could you tell us why not?

Adriana: Oh, true. Yeah. Yeah. So, um, you know, as I mentioned to you, when we spoke earlier, I'm not a Prometheus person. I'm not used for me to use. But when I was like, you know, working, when I was managing this observability, observability team of Tucows, and, you know, I think Prometheus was was part of our stack. And I started digging around like I, I'd heard things from from folks saying like, Oh, Prometheus is like really hard to use and manage. And I'm like, yeah, doesn't sound like something I want to be, you know, dealing with. So I thought, well, maybe there's a way to, maybe there's a way of not using Prometheus. So when I started putting together this, the stack for for use of Tucows, my thought is, well, if we can do away with Prometheus, that's like one less thing that we have to manage. What, but how can we make that happen in in, in like a realistic way? And Fortunately, I think OTEL comes to the rescue for us in a couple ways. So first of all, there's a component of the OpenTelemetry collector called the Prometheus receiver and just for a bit of background, the OpenTelemetry collector is basically a vendor neutral like binary or agent that is used to ingest, transform and export your data. So it'll ingest your data from multiple sources, whether it's metrics, logs, traces, either from your applications, infrastructure, et cetera. And then it's basically like a data pipeline, right? It can manipulate your data and then poof, send it off to like one or more observability backends. And so the collector has this component called the Prometheus receiver, which basically can ingest Prometheus style metrics from your infrastructure. Because as I learned as part of my explorations, Prometheus isn't just a data pipeline. a tool, it's also a data format. And so I thought, oh, this thing can ingest that data format. Well, if it can ingest the data format, and then we can send it to like some other back end that can process, you know, do something useful with these metrics. And I don't need Prometheus then. why run Prometheus? And I think like, and it's funny because at the time, like just as when I was, you know, really promoting OpenTelemetry at Tucows, OpenTelemetry itself was not like, it was fairly nascent still, and this Prometheus receiver was probably even more nascent. And so it's definitely not in a state where we could, you know, fully replace Prometheus. But I think we're starting to get to the point where there's like a lot of the tooling around OpenTelemetry is going to enable that. So we can basically reduce our stack, right? Because again, like, I think the real dream is just having this all in one platform. Why do we have to manage like even the SaaS vendors? Why do I have to manage like... so many SaaS vendor tools as well, right? Because there are companies who like, you know, they're running OpenTelemetry and they're sending like traces to one place and their logs to one place and their metrics to another place. Like, so you are paying a crap ton of money for all this stuff. Like, let's see if we can reduce that.

Bart: I think a lot of people would be looking forward to the day when that unification happens, when that simplification happens. Until then, I'm going to bring this back to Hannah. Hannah, you've written a lot of subtitles. What's SaaS?

Hannah: I have no idea. Usually, someone says something. You don't want to mix something up? Oh, well, okay. San Francisco articulating Subdivision System.

Bart: Wow. Even a different acronym than the one we're accustomed to. I love this. This is folks you're witnessing innovation in its purest form. Great job, Hannah. Hannah, you're going to be back on this podcast whether you like it or not. This is, you know, we got to play buzzword bingo with Hannah at some point. This is really, really good. No, but I hear you on that. One thing I do want to ask you though, Adriana, is that particularly someone being in positions of leadership. When you are faced with choices that are generally nascent technologies and you're looking at a team and you know the maturity level and you know where they're at, you know the things that are gonna be difficult, how do you approach the topic of saying, hey, this is the direction we're going in, these are the technologies we're gonna be using, there's not necessarily great documentation and support, how are you gonna help them through that process of learning the frustrations of the onboarding? What was your experience with that and what advice would you give to other folks that are helping teams upskill?

Adriana: Yeah, that's a really great question. Because, you know, I faced uphill battles, like both from the individual contributors and and like leadership alike, when I was at Tucows. Because because of those things, right? From the individual contributors, it was like, well, we're using open tracing, which is one of the precursors to OpenTelemetry. So isn't that good enough? And I'm like, but then tracing was like retired. So that's not going to do you any good in the long run. And then at the same time, like me going around saying, well, you know, let's, let's use OpenTelemetry and saying that to leadership and trying to convince them to go that way was was was tough because of the, well, this thing's not mature. Like, how do I know this is gonna do what I need it to do? How do I know this thing's like here to stay? Right. And so one of the things that I did, so we were, we were We were, even though we had like a, we were paying a vendor at the time, um, for our observability stack. Um, I wanted to also make sure, well, okay, how was, how was this vendor selected? Are they in fact the right fit for the organization? So I was, I, I had contacted a couple of other vendors, um, to kind of get, um, to see like. how do you approach observability, right? Because it's really like in an OpenTelemetry world when everyone's like speaking the same language, what differentiates one vendor from another is really how they render the data, right? And so we're speaking with these different vendors and getting them, and they were like super supportive of OpenTelemetry. So it was one of the things that I did was like, okay. Can I get reps from each of these vendors who are working in the OpenTelemetry community? And you come under not the vendor umbrella, but the OpenTelemetry umbrella to come speak to these folks at Tucows to basically answer any of their burning questions and to sort of calm their nerves, to let them know this thing is like, it really is legit. It's here to stay. To my surprise, I... got two very influential people in the community, Liz Fong-Jones and Ted Young, to come and talk to the developers at Tucows. And I'm so grateful for that because they came there just to answer questions, address any concerns, and it was really, really wonderful. And I remember turning to my team at first. I'm like, okay, we need to like... we need some like seed questions in case there's gonna be crickets for an hour. Like, oh my God, I've organized this thing and like, what if nobody has questions? So I'm like, I need questions. And, and you know what? Like the, uh. the developers within the organization had tons of questions and we filled up that hour no problem. So there was definitely a desire to want to embrace that. And I think having sessions like that, and I know that's not always possible to do. I mean, honestly, I feel like I should have played the lottery that day to have been able to get that kind of access. Because I know, especially as the project gets busier and busier, that kind of access is definitely a lot. a lot harder, but to be able to have folks come in and have, you know, both our engineers and our executives alike see that kind of session, I think helped bring a lot of confidence into the project. And so it's a lot easier for them to like warm up to this idea of OpenTelemetry.

Bart: getting the right people in the room, showing them they're not alone, getting those questions answered. I think that's a fantastic way to build confidence and also additional curiosity and showing them where they're at We are now in time for the great CloudNative Bake Off. Hannah, this is where you take over. We give you the keys and you drive, even though you're not old enough to drive yet, because I don't think you can be 15 and drive.

Hannah: Yeah, I can't drive yet. Hopefully soon though, you know?

Adriana: All right. Next year, next year. This time next year, she'll probably have her license.

Hannah: I really want to be able to drive. Like, I don't know. There's just something about wanting to drive.

Adriana: Meanwhile, I have a license and I avoid driving.

Bart: I hate driving. I try to avoid it. I think it's in Spain. I don't have a license. I do in the US. So I drive when I'm in the US and then I don't drive at all here. But Hannah, we understand that you recently spoke at a CNCF meetup in Toronto. Can you tell us what the topic was about? And what brought you to do this? What was that experience like?

Hannah: So, um, The crickets that you were talking about, they came to this. So, we did, I did a talk with my mom, my mom, my mother. Yeah, so, well, because it was like a techie thing, and I have no idea how techie things work. We wanted to do something that drew parallels between my interests and her interests. We had previously recorded like an extra thing on the parallels between baking and coding. So we decided to turn that into a talk. So we talked about how troubleshooting is very similar when it comes to baking and coding and how overall the process is very similar and you have to do all these things like experimenting, refining. And overall, we just kind of talked about that.

Bart: That's great. I mean, just one thing that might come to mind there, and I think it's wonderful to draw these parallels. And I think it's as a broader thing to with anyone that's in the cognitive ecosystem, being able to transfer how these processes work, whether it's coding, whether it's building infrastructure, whether it's observability. I always say is that until we are able to put these in real world examples or metaphors, I don't think we're doing our jobs well enough because it even gets to the point of gatekeeping. It's like, no, we're gonna keep these things intentionally complex so that they're only accessible to an inner circle. I really like that idea of drawing those parallels. I'm just curious, you know, because there is a lot of creativity in baking as well as in programming. encoding what were some of the parallels that you put between the two of them.

Hannah: Well, mainly it was the troubleshooting one, because so I had a lot of trouble making macarons because they're very difficult. And so we talked about the process of that and how it wasn't repeatable and how we didn't read the recipe at first and how, and then she kind of brought the code thing about how, well, sometimes you just like go into it and you think you're an expert and you just start writing the Cody things. Um. And so that was our main parallel that we drew between them.

Bart: Great. Anything you'd like to add to that, Adriana, about your experience giving the talk together?

Adriana: Yeah, yeah. So it was lots of fun doing the talk and really like, as we put the talk together, really seeing like how many parallels there were. And I think like one of the ones that we mentioned, for example, is the fact that you have to do a lot of research, right? Whether you're having to like Google, you know, articles, blog posts, videos, which is Hannah's favorite way of consuming information. or even talking to experts, right? Whether you've got a connection on Slack, right? That you can hit up or like in Hannah's case, like hit someone up, like phone a friend or ask a question on Instagram Messenger or whatever to just to get that extra little bit of information to help you with troubleshooting. And then as Hannah said, like making sure that, you know, you nail a recipe or you nail your code, but it's not repeatable. that's problematic, right? So you run your code like 10 times and it's fine. And then the 11th time it's cracking out, well, then there's a problem. And it's the same thing with your baking, right? You follow the recipe and it works. You know, we had a picture where we had like this beautiful bake, this beautiful macaron, and then we went to do the recipe again. And it didn't work. did not turn out well. So obviously there was a problem, so we needed to make sure that we got it to a repeatable point before we could then start refining and putting on the bells and whistles.

Bart: I love that. As a long time baker of chocolate chip cookies, it's the only thing that I bake, but I'm a big, big fan of making chocolate. I only make them when if there is an occasion. And so like last week it was, one of my partner's friends is pregnant and she was at a particular craving for chocolate. And I was like, hey, happy to do this. And it's something, it's like I said, it's the one thing that I do, but it's something that over time, that troubleshooting and, respecting the steps in the process and the measurements. Going back to what you had said in the very beginning of the conversation, Adriana, is like, if you are going to learn a technology and lead a team based off two tweets, probably not a great idea. Just like if you're going to make a recipe off watching one 10-second video, probably going to be an interesting result, but not necessarily the result that you desired.

Hannah: I've done that multiple times. Okay. I don't like following a recipe. I hate it. Well, okay, I don't like being told what to do and a recipe basically tells me what to do and I'm like, just because it's written down doesn't mean I'm going to listen to you.

Bart: I agree. I could not agree more, Hannah. And as I see that recipe, I'm like, well, this is one person's opinion. There will be many others. And my opinion is also valid.

Hannah: Yeah. I judge a recipe on how many eggs it has. Cause I'm like, I'm not using that many eggs. I'll make up my own recipe.

Bart: This is great. So I really, I really liked the parallels though. And also that when the end result then makes you question whether it's good or whether it's not what you wanted, What were the things that could have gone wrong in the process? Was the oven on the right setting? Did I use a different brand? Brands change everything. If you go to another country, my poor mother, when she came to Spain two times ago, she tried to make this kind of like pancake. And it, but it's because of the ingredients that she has in the United States, the flour, the water can influence it depending on which country you're in, the salt, all those different things. can play a huge role. So you're like, no, no, I'm following the exact steps, but the result doesn't come out how you want it. So then you have to ask where did it get, where was that mistake? Where do you think is going wrong? I think that's a brilliant parallel between the two. So that's wonderful. Good. We are unfortunately getting towards the end. I could keep talking for quite a while, but I want to know for both of you, Hannah, let's start out with you. What are your next steps? Do you want to continue editing podcast videos, writing subtitles, coming up with new definitions for tech terms? What do you want to do?

Hannah: Well, I really do enjoy coming up with new definitions for tech terms. But I definitely want to continue on doing the editing. I also enjoyed doing the graphic design aspects of it. And overall, I love doing anything that has to do with marketing and graphic design and just being creative in general, coming up with new ideas. Um, so it might be something that I go into. I'm between that and dentistry. Um, dentistry is a bit higher, but like, I think this is a fun kind of thing to do on the side, um, after school and just something to build experience because maybe it's not the thing that's going to turn it into the biggest break of your life, but it's an experience and you're going to take. memories and not only that, but skills that you've learned from that and be able to turn it into something that you will make your big break.

Bart: Wow, that's a beautiful summary. And at your age, I was not saying anything that interesting. I would encourage you to continue with the graphic design with all those other elements that can be used in so many different areas. I would not recommend to start doing dentistry in your spare time. Yeah,

Hannah: I'm definitely not doing that in my spare time. I'll wait until I'm licensed to actually do that.

Adriana: She can spew out great dental facts, though.

Bart: That's good. That's really, really good. And it would also, once again, be interesting to see the parallels between those different, because I really think that's where these things come to life. And you Adriana, as someone that's actively sharing knowledge I think that creativity comes in there to try to better connect with your audiences. But what do you have on your agenda for the coming months?

Adriana: Oh boy. So I think the thing that's been really like that I've been working on the most in the last little while is I have an O'Reilly video course that's coming out in 2024 on observability and OpenTelemetry. So that has been most of my summer. And then I will be at KubeCon. I've actually got a couple of talks coming up, one on like part of KubeCon proper in the platform engineering track with a little bit of OpenTelemetry sprinkled in, which is always awesome. And then I've got a talk also for Observability Day, one of the co-located events in KubeCon, which is about the observability of CI-CD pipelines. So really looking forward to that. And I think next month, no, this month, oh my god, in a couple of weeks, I'm going to be at All Things Open doing like a panel talk with some of my co-leads from the OpenTelemetry End-User Working Group. So really looking forward to that.

Bart: Action-packed, didn't expect anything else. Looking forward to meeting you in person in Chicago. And Hannah, we're going to be seeing you doing lots of fantastic things no matter where you are. So I'm not worried about that at all. Just keep doing what you're doing. It's been a wonderful conversation and we'll definitely have to have a part two. So don't go too far. Perfect, thank you very much. Cheers. Pleasure.

Adriana: Bye.