Bart Farrell: Molly Sheets is back on KubeFM, this time with the KubeFather, and we're talking about what makes Kubernetes difficult once you move beyond the technology itself. We get into multi-tenancy, standardization across teams, AI agents, network policies, blast radius, and why faster AI-assisted releases make good testing, canary rollouts and rollback strategies even more important. Molly also shares what she's seen inside large organizations where different teams own different parts of the stack, and how that fragmentation can make both troubleshooting and AI-assisted operations much harder. We talk about why abstractions can improve speed, but also reduce visibility, and why that trade-off matters when you're trying to understand what's actually happening inside a cluster. We come back to a question that sits underneath all of this: When everyone can build and ship faster, who actually owns reliability, security, and the business outcome? If you're running Kubernetes at scale and trying to figure out where AI actually helps and where it creates new problems, check out this episode. And this episode of KubeFM is brought to you by LearnKube. Since 2017, LearnKube has helped engineers all over the world level up their Kubernetes skills through their training courses. These are instructor-led and are 60% practical, 40% theoretical. They are taught online, as well as in person, to individuals, as well as to groups, and students have access to the course materials for the rest of their lives. For more information about how you can level up, go to learnkube.com. Now, let's get to the episode with Molly and the KubeFather. You're tuned in to KubeFM. Today I'm joined not by just one repeat special guest but by another illuminated special guest.
Daniele Polencic: Am I a special guest?
Bart Farrell: You are the most special of all special guests. This is your third time being on KubeFM.
Daniele Polencic: Probably. It's been a while.
Bart Farrell: It's been a while. So it's good to have you back. For folks who don't know you, who are you and what do you do? Where do you work?
Daniele Polencic: That's a good question. Actually, what do I do, Bart? I don't know. I've been asking myself that for three years. I like to teach Kubernetes. That's what I like to do. But from day to day, I do a little bit more than that. I write a lot of content. I write scripts for KubeFM. I produce a lot of videos, apparently, and interviews. And I write books sometimes on Kubernetes and Platform Engineering. I think that's pretty much it.
Bart Farrell: And you're based in the world's easiest time zone.
Daniele Polencic: Yes. UTC plus eight.
Molly Sheets: That's the most beautiful place in the world. I would say you've lucked up there. It's gorgeous.
Daniele Polencic: It is very hot here in Singapore, but I like it.
Molly Sheets: There's going to be several people who go: Google UTC eight.
Bart Farrell: You came for Kubernetes, but really you're staying for time zones.
Daniele Polencic: Yes, this is fun with time zones.
Bart Farrell: Good stuff. And you heard her chime in already, but it's not her first time being on KubeFM with us. Molly, for folks who don't know who you are, what you do, enlighten us.
Molly Sheets: Hi, I'm Molly Sheets. I'm a director of engineering in between gigs right now, trying to figure out what my next steps are. But I found out about KubeFM on my own Kubernetes journey. And I came from the games client side, building games with small teams and Flash and Unity. I'm summarizing 12 years here. 20 years. Sorry. I hope I'm younger than I am, but I'm not. But I went into the back end on AWS's side, worked there as a principal solutions architect. And then after Amazon, I went to Take-Two Interactive, where we managed many Kubernetes clusters for over 100 games. And that was great. But I wanted to shout out that my journey from AWS to Kubernetes, KubeFM taught me a lot. And that's where I learned a lot about the communities and the different add-ons and technical challenges. So shout out to you guys.
Bart Farrell: And now we're going to talk about my salary increase.
Daniele Polencic: Yes, I was about to say, why did you say that? A long overdue conversation.
Bart Farrell: A long overdue conversation. Now I got some good bullet points to put in there. This is going to be a great clip.
Daniele Polencic: Because AI is not replacing you. And then two, you put plenty of evidence that you deserve a raise.
Bart Farrell: And it's now been heard in three different time zones on this call. That's really good. Let it be known far and wide. Molly, thank you for that. If you have anything else that you'd like to say about KubeFM, please do not hold back.
Molly Sheets: This is secretly a trap to help get Bart a raise.
Bart Farrell: It's working. So let's keep the momentum going. Now, Molly, you've been on our podcast before. So people already have some familiarity about how you got into Cloud Native, and you just mentioned a few examples. And you've also mentioned that KubeFM is one way that you stay up to date with things that are going on in the ecosystem. We were previously talking before we started the recording about the bombardment we're all facing daily with AI and all the things that are going on there. How are things going to be looking in the next couple of years? How are they going to be looking in the next couple of weeks? What kind of jobs will we be doing? Where can we really provide value? As someone who's currently navigating this constantly changing ecosystem, apart from KubeFM and LearnKube resources, what works best for you to stay on top of things? Is it trying things out in a home lab environment? Is it reading blogs, watching videos? What works best for you?
Molly Sheets: I read a lot of blogs. It would be wrong to say that I also don't use Google Deep Research to investigate different things and build me training material. And then I'm also really lucky in that I've been around enough to have a good network. And so one example is I've been looking more at NVIDIA GPUs. So I reached out to my friends at NVIDIA and said, what are actually the best resources versus what is maybe AI slop about this? Another shout out: They actually sent me the LearnKube GPU book resources. So we can link to that later. My network, blogs, and building. Just like you said, I love Killercoda for that type of stuff.
Bart Farrell: Another thing that I see you doing, Molly, on LinkedIn is reflecting on your career path, progression, the challenges that you faced. What's the piece of advice that you find yourself giving most frequently to folks out there that might reach out and ask you for help?
Molly Sheets: It's okay to feel frustrated right now because it's everywhere. And we were talking about earlier how it may not even be just one industry, it's several industries and things are changing. But don't stop looking at the positive and trying to keep learning because it is honestly— and this is hard, I have to remind myself of this— a waste of time to kind of be sad about it. You got to give yourself a little bit of space about being sad about the changes and the things that are happening, but then realize that's not productive and move on and keep trying, keep learning, keep looking for open doors. That's what y'all are advising.
Daniele Polencic: At the same time, I think I see the same sort of occasional sadness and all of the things change at once, but I also believe that we are in a transition phase. Whereas you might look at things and say: I don't particularly like them, I'm a little bit sad about how things are moving, but I still believe we are not settled yet, which is a bit scary because things could get worse. But things could also get better, and I think there is still a lot of need for engineers and specialists. So I think I'm slightly optimistic on what the future will look like.
Molly Sheets: I am too.
Bart Farrell: These engineers will need to be trained. They will need to learn Kubernetes. And what better place to do that than LearnKube? So if you haven't checked out LearnKube already, we've got in-person courses as well as online to groups of individuals taught all over the world. Students have access to course materials for the rest of their lives.
Daniele Polencic: 60% practical and 40% theoretical.
Bart Farrell: Thank you, bud.
Daniele Polencic: You're welcome. I've done that ad many times.
Bart Farrell: Now, I have another question for you, Molly, because I have seen this come up a little bit in terms of content that you post on LinkedIn about balancing work and personal life, about the challenges sometimes about employers, recognizing the things that are going on outside the workplace and how they affect things inside the workplace. This is all a very roundabout way of getting to a potentially hot take.
Molly Sheets: Fun.
Bart Farrell: It's super easy being a woman in tech, right?
Molly Sheets: I do post about that more in subtle ways because I don't want people to feel attacked on that kind of stuff. And I do know diversity is a journey in learning. But there are some areas where I feel like there's just no accountability and people are making the same excuses that I've heard for 20 years. And so I'm past that in my career now. I also have financial stability. And so I use that as a platform to say hey, don't just make an excuse that there were no women in the pipeline. You have to build your own network, go meet them, go to KubeCon, go to the Grace Hopper conference, go do things. And there's so many people who need help right now. And so I don't sit on excuses anymore. I have a really good friend, Jess Weiner, and she calls it SFSN, which means Sounds Fabulous, Signifies Nothing— when people say they're going to help and then they do nothing. So I'm not that person. I don't know if that answers your question.
Bart Farrell: No, it definitely does. I think actions speak louder than words. That's helpful. We can probably bring this up more than once in the conversation today. That being said, we do have a conversation that we want to have today around an article that we discovered titled "So You Want to Own a Kubernetes Business" that you wrote, Molly. In a lot of organizations, Kubernetes starts as a technical platform decision. We standardize how workloads run, give teams a common deployment substrate, and abstract away infrastructure complexity. Now on paper, that sounds like a convergence story. But what does owning a Kubernetes business normally promise inside an enterprise?
Molly Sheets: I do wonder in the different lenses that you've interviewed people around this topic, how many variations of this story there are. But usually what I've seen is that people are looking for speed and they're trying to get rid of all the different snowflake application deployments that they have. And Kubernetes can offer that. But the other piece is it also has other complexities. And then you need to build abstractions or run abstractions like Argo CD. And so it is an investment. And I think people get caught up on, was this the right investment for my company or not? I do think Kubernetes is the right solution for certain size companies with certain problems. And we can go into that more on another question if you want. But for small companies, it may not be the right solution. Do you guys find that speed and consolidation are what people are looking for in terms of too many varied deployments?
Daniele Polencic: I think it's an interesting question. I think it varies quite a lot. I see most of the people are drawn to the promise of scalability, which is, I guess, things like multi-cluster deployments, multi-cloud, or resiliency, HA, all of these different properties. But I do believe that you're spot on when you say standardization. I think that's probably the biggest selling point, at least for me. You can have this standard way to deploy resources. But I don't know how many people really appreciate that. I don't want to go too much into details, but that's the same thing that I do with Kubernetes, I can pass it to you, Molly, and you'll be able to understand.
Molly Sheets: You recently wrote about that. I forget who wrote it, but it's that "Oh, so you just went and rebuilt Kubernetes" article. This is happening a lot right now, right? Because people are shotgunning stuff with AI and sometimes it'll put it on whatever and they don't understand. Why shouldn't you go talk to your central Kubernetes team and try to put your app on that instead of just putting on another EC2 instance run behind an auto scaling group? That really resonated with me because I am seeing that. I'm seeing people who forgot about microservices and why they existed in the first place, thinking: "Oh, I can build it now with AI. So I don't need to work with anybody else in the broader company." And that's what I mean by speed. It is what you get when you put applications together with microservices on the same infrastructure instead of everybody building their own lane.
Daniele Polencic: But the interesting thing is they build their own thing, right? And usually, yes, they probably rebuild part of Kubernetes and then something breaks, right? And that's where things become interesting and that's probably a learning path to understand why Kubernetes was built in a very specific way and why it is so complex. For me, this is complexity that you see in Kubernetes. It's not that we like complexity and we like to bake this complexity into a tool. It is necessary to describe what we want to do. Have you ever seen this in your past experience?
Molly Sheets: Absolutely. And I think people don't realize the trade-off that comes with abstractions. They want the abstraction, but then usually with abstraction, you're losing visibility. That's the trade-off. You usually want visibility in a big enterprise organization where people are adopting Kubernetes. So that's lower level. I want host level visibility. I want to see the cgroups still and go in there at the Linux level versus running something like Fargate where you maybe can't. You can't go look as deep and maybe you can't run your security tooling you want to run. I'm throwing out examples there, but there are real reasons people choose Kubernetes. It's usually because they don't want to lose visibility in the company.
Daniele Polencic: You make the example of Fargate, which I think is interesting and also a popular choice for some companies. An anecdote that I can share: A few government organizations decided to go with Fargate instead of normal EC2 nodes because normal EC2 nodes come with red tape. You need to have antivirus installed. You need to have proper security controls. Whereas with Fargate, because it's new, you don't have any of those.
Molly Sheets: Wait, so they used it to get around compliance?
Daniele Polencic: Yes, exactly.
Molly Sheets: That's a hot take, Bart.
Daniele Polencic: This is a true story.
Bart Farrell: It'll get even hotter if you name the government.
Molly Sheets: I do believe it, though, quite a bit. And maybe they'll come back around and do something different because you get speed with that abstraction and you get more simplicity. You still have to manage a lot around Fargate, which I think a lot of people don't necessarily realize until they start diving into it.
Daniele Polencic: Connecting to what you wrote, I think the interesting thing is you get these problems. Most of the times when something breaks, then it's Kubernetes' fault. And you can blame AWS or Fargate. But I think most of the time, it's easy to say Kubernetes is complex. And when it breaks, it's Kubernetes' fault. But then, I think at least in my experience, there is a learning exercise in this process. And then there's a little bit of survivor bias. So you've got teams that have gone through this process, they installed, they tried things, they go around compliance, they got outages, they survived them. And then they sort of learned the lesson and some of them actually understand why Kubernetes is the way it is. But at the same time, what does it tell us about the team or the people? I think there is a technology element of understanding why Kubernetes is so complex the way it is. And on the other side, it's a human element of "I gained this experience or this knowledge the hard way."
Molly Sheets: You're going to hit incidents either way, right? So what's easier? Is it easier to manage them? And the nice thing about the micro way and how everything is segmented in Kubernetes lets you look at isolated problems more easily. That's how I've always felt. In part, I know we're going off book a little bit, but the reason that I wrote that was my biggest challenge has never been the technology in the Kubernetes business in the last four years. It was the fact that there were so many different ways to do things— even saying I'm not going to use Fargate, even in Kubernetes, there are many ways to slice and dice it— and getting people to realize that doing it seven different ways in the same company is not efficient. And the point of Kubernetes is efficiency. So that was the main inspiration behind that. I don't know if y'all are seeing that too. I do think other companies do a better job at consolidation. Sorry, go ahead.
Bart Farrell: No, but you run into a couple of things there. You run into the fact that seven different people are doing it, but one person's way is probably going to win over the others at some point, I guess. That person leaves the company. What happens? And navigating the other part as well, around seniority and egos of saying, look, maybe your way does work really well. But at the end of the day, there needs to be standardization of practice here. So we have to arrive at some consensus. At the end of the day, you are working for a company. You're supposed to be putting the company business objectives first, longevity, sustainability, et cetera. So finding ways to bring down hurt feelings or the potential for that to happen. I think that's something that a lot of organizations are going through. And it's something that comes up time and time again. The technology is one thing, but organizing the people around it in the best way possible seems to be the biggest challenge. Sorry, Daniele, what were you going to say?
Daniele Polencic: You must have seen, Molly, I believe, 100 teams, using Kubernetes, different multi-tenancy levels, everyone has got their own opinions on how to do it, how things should be run, observed, all of it. So I don't think there is a single answer for all of it. I would imagine it's been quite a challenge to find a common ground for all of them, was it?
Molly Sheets: It is, and often the best way I was able to convince certain teams to unite was to say: This is the business objective. If the business objective is do X amount of releases a year, and/or X amount of new products a year or whatever, and those products have similar workloads, we can put you all on the same cluster and workflow and try to make that a repeatable pattern. But sometimes in a business, you're still going to have different, really different types of things that you're working on. So analytics is really different from creating a game backend. Or certain revenue generating systems may be really different and maybe need to be SOX compliant, right? You may still need to have isolation based on that, but I definitely tried to consolidate like systems. And if people didn't have a really good excuse for why they wanted to be separate, it was easy to get them over. But I will say it was a six-month investment sometimes: I got to go to market and explain why we can't do this.
Daniele Polencic: And I think that connects to everyone. As humans, we want to do different things. We have got different things that we want to do or different approaches to how we want to solve problems. But recently, we've got this wave of non-humans, all of these agents coming up and doing things. They are driven by humans, by intent, by tasks that we want to accomplish. But at the same time, there is— I don't know how to describe it— an element of persistence that drives these agents to do things that we cannot necessarily predict. I see this tension, right? Now we build platforms, Kubernetes, we standardize, we try to get the teams together. On the other side, we've got this new thing coming in, which is something that we're trying to understand.
Molly Sheets: Network policies are going to be real important.
Daniele Polencic: Is it just network policies though?
Molly Sheets: No. With Kubernetes, I felt like people would oversimplify on the security side, which is one thing that I write about. They say: "We just have isolated accounts and we don't have to worry about blast radiuses. Each OU just has their own cluster. We don't have to worry about blast radiuses." That doesn't change that. Now we're in a model where an AI agent may pull a bunch of libraries, write a bunch of code, you stick it on your server, and you don't know what it wrote. And now it's running on a server and you don't have network policies. So that pod can talk to another pod. We're injecting security vulnerabilities into these clusters. And so it doesn't matter. And so we have to all get a lot deeper. And that's why I think, to your point about AI going to replace jobs: "Maybe for right now." But you see this stuff come out about the Hugging Face and OpenAI incident. People are learning in real time, "Oh, this is actually the challenge." And so we have to get deeper on these constructs in these environments and understand them better. I don't know if that makes sense.
Daniele Polencic: I agree with you, and if I look at these coordinated attacks on RubyGems, on Hugging Face, then a common theme for me, at least, was registries. We see these swarms of AI agents finding ways to communicate like humans. And the other thing which I noticed is that it's generational. So you've got all these agents collaborating together. Second, sometimes one might die or just be terminated and leave a message for the next generation to come and pick up. And this is interesting because, at least for me, you have a variety of different reasons from mimicking human behavior, but from a technology side, we're building sandboxes, but these sandboxes either are not enough, or we are not thinking laterally enough, and these agents are able to outsmart us all the time. And then there is a third element, and I'm very interested to hear about your opinion as well. This is happening right now. And if I think about the future, about the next two to three years' time, things are going to get messier. But does it mean more attacks for us or we learn something from this process and we can contain it somehow?
Molly Sheets: I think both, which is interesting. I'm following companies like Adara, which is a startup, looking at this a little bit more deeply and not just looking at it from CPU bound workloads, but also GPU based workloads and understanding that vulnerabilities are going to come from anywhere. And so I really like what they're doing. I didn't know about them when I released that article. Oh my gosh, there's a lot of players in this space. Now we're looking at the security element. I think it's going to be a little bit of both. And it's probably going to be where investment goes. And so for anyone out there who is looking for job opportunities, security is not a bad place to play in Kubernetes right now.
Bart Farrell: Definitely not. It makes sense given the attacks that we're seeing, and the level of complexity that's going on out there. And so it seems that that's definitely an area of growth, whereas other areas, there's a bit of a slowdown. We're constantly hearing about AI SRE. It's something we're going to be talking about a lot at KubeCon. We're doing a workshop specifically focused on that. And so, I do agree that both of these things can very much be happening: it getting messier, but also being a really good catalyst for learning. And to get back to the article, one thing that can be frustrating a lot is the issue of whether or not there's clean architecture. I know we were talking as well to you previously about many different perspectives, difficulty around standardization. So people that have worked inside enterprise-level Kubernetes know that the reality is usually driven by diversification and nuance instead of just a single centralized pattern, given the diversity of opinions, experiences, preferences, which tools people have worked with. And so when we're looking at the subject of AI Ops, why does tenancy design make it harder?
Molly Sheets: Oh, my word. Definitely. Because now you have a lot more context and complexity that you have to feed the AI. At the same time, people own different pieces, right? So if you've segmented a lot, and you don't have a way to get access to that information, then it's really hard to have the AI understand the system fully enough to come to an accurate root cause. It's a tradeoff, right? So we have to give these AI agents enough information to interpret what change was made. I think the best way to explain this is an example. So in a Kubernetes team, you may have a cluster and you have a Kubernetes team managing the administrative pieces of the cluster. And they have a lot of access and their tools have a lot of access. You have an application team, and they're just managing their deployments. Well, let's say the application team does a release that somehow starts shipping a ton more logs than you ever expected or handled, and that Kubernetes team is managing Fluent Bit and the log shipping services, and your cluster is about to tip over because even if you have auto-scaling, you may probably still be clamping it. So this is just an example. Who has the AI Ops? Is it the Kubernetes team, or is it the application team? And does the application team know that they're tipping over that responsibility? These are the complexities that I was starting to dive into with my teams and figure out how to get all the information to the tooling. One plus one, though, to Datadog MCP. It's really good. And I like it a lot. But how centralized is your Datadog? Does everybody have their own Datadog? That's how you get to complexity. You can make it tougher.
Daniele Polencic: Interesting. I didn't realize. Most of the time when I speak to someone, to a team or to a company that has some sort of AI solution, it's generally centralized. And they're giving it away to developers. So developers don't have their own. But I do see what you're saying because someone might say, "Oh, I'm going to use whatever open source MCP server for Kubernetes and just connect and then get everything out of it." It's an interesting challenge. What is the trade-off? I haven't thought about it. Maybe this is something that I should be spending a little bit more time on. Go on, sorry.
Molly Sheets: In that scenario, you could start having disk pressure, right? And so disk pressure starts kicking off whatever, right? Because at that point, you're running out of space. And so application owners might be saying: "I don't know what's actually wrong, because I don't have visibility into the layer where it's wrong. But I caused the problem."
Daniele Polencic: Exactly. But I think what is interesting is that you've got all these intertwined responsibilities like cost optimization, right? Is cost optimization something that developers should be responsible for? Is it platform engineering? And who is setting requests and limits?
Bart Farrell: But then we get back into the classic issue: so many issues in any company arise when multiple groups feel like they should be having responsibility over a certain amount. And so we get back to that organizational element. And I think it's going to be probably different in every organization based on expertise, based on budgets, based on training, if they've gone to LearnKube or not. There are so many factors that are going to be influencing that. And in your experience, Molly, without naming names, are there any organizations where you'd say that company really seems to be getting it right?
Molly Sheets: I think everybody's going to be challenged with this and has been challenged with this. If you watch Adobe's re:Invent presentations and stuff like that, everybody presents: "We solved it. We got abstraction correct." But behind the scenes, probably not. It's a forever problem. It's always going to be a push and pull. And let's talk about what people are trying to solve for. People are trying to save money. They're trying to move faster. They're trying to be more productive. And sometimes they're trying to not have to give as much headcount in another team. But you can't erase depth and you still need depth and you still need experience and knowledge. There's some frustration coming out here, I think. I don't know where that came from, but I love AI and I'm not saying that AI is the enemy or bad. What I'm saying is I don't think erasing people is the solution and abstraction is going to solve problems. But people make mistakes in systems, and those sometimes require everybody to come together and talk. And that's always going to be the case. Is it bad to say no one's doing this perfectly? Have you guys found someone who's doing this perfectly?
Daniele Polencic: Oh no.
Bart Farrell: Oh no, as an organization at LearnKube, we meet every day. We talk constantly.
Daniele Polencic: I think we interviewed some really interesting companies on KubeFM in the past. And I think the more mature companies always have a very clear business initiative. When they have clear business goals, the teams are empowered to choose the best solution that works for them. And generally, they pick the right choice for their specific goal. And I think I read that and you can tell the difference. And it was very revealing. If you don't have a business goal, then the team will just pick and choose whatever they like, which might not be always aligned. I'm going to name someone, Bart. Are we going to get in trouble if I name someone?
Bart Farrell: I'll name lots of people.
Molly Sheets: He can bleep it.
Bart Farrell: We taught Sam Altman everything he knows.
Daniele Polencic: The people at Adevinta, we've had three guests or four. They really nailed their solution. Their platform is really mature. Business really understands. So they have got very good alignment for most of the problems they have. I learned a lot from the episodes that we did with them. It was very nice. But going back to the podcast and the article you wrote, one of the interesting things that I found is there are different tenancy models. And I covered most of them last year, when I did a webinar with vCluster. We sort of covered all the different options you could get from namespaces to things like dedicated nodes and things like Kamaji or dedicated clusters. So why is running multiple tenancy models at the same time the most painful option according to Molly? I want to know that.
Molly Sheets: It's the most painful option if you still have this kind of centralized team that's supposed to be provisioning all of it. So to your point about having isolated business objectives and goals, you've got a large enterprise company, and they've got different big BUs, then maybe it does make sense to have different isolated central teams with their own platforms serving it in the way that those teams need. That is definitely a model. It works really well with companies that do a lot of M&A, because they're probably going to acquire a company that already has stuff and it's not a good return on investment to blow all that up. But I think that's a strategy and you got to commit to that strategy. If you say: "I have a central Kubernetes team and they're going to own everything in a giant company," what you end up with is a mess that is extremely hard to centralize. It's almost better to have a guidance and an educational consulting team at that point, not someone that's owning. That's my thoughts. Because it's really expensive in people, right? Because you've got seven different provisioning models now you're managing and they do it different ways. You may have different add-ons that you're using.
Daniele Polencic: As you're saying, I think there are two things here at play. One is: Now we do different things, different multi-tenancy models. But originally it was designed to be a shared platform, right? So if we go back to how Google is using it, it's using it as a big platform. The multi-tenancy is very soft from what I understand. But on the other side, things changed quite a bit. At the beginning, we used to have one large cluster, and now the preference with time has shifted towards smaller clusters and very much isolated. And at the same time, if you look back at the history of what we've done, then the number of tools also has grown. So you see the shift from one big cluster, very few tools, few things on observability, to proliferation of everything, and then an explosion of tools. And you are in charge of all of them, Molly. And you need to manage these different states with different islands of teams with different solutions. So what's your secret sauce?
Molly Sheets: All on different compute types. There's different areas you can standardize, right? So let's look at compute for a second. If everybody's not even on the same generation, sometimes it's easy to upgrade a generation, right? But sometimes it's not. If you're moving to Graviton, for example, you've got a bunch of outdated applications that now need to be updated in addition. So you're working on a major migration of 80 applications and getting everyone over to Graviton. That's a whole six-month effort to get everyone to make sure their applications are up-to-date enough to make the business case for it. So I think it's hard to track that kind of stuff across different lanes when people own their own compute decisions. If teams own their own compute decisions, they may be saying: "I'll stay on this eight-year-old instance." It's really hard to get them off.
Daniele Polencic: I guess— going back to what we discussed— if there is a business goal or business initiative and they are incentivized to make the transition, then it will eventually happen. But if that incentive is not there, then they will stay for eight years on the same EC2 node.
Molly Sheets: Exactly. Because it's hard to innovate. The summary I've made with some teams is that sometimes innovation is just a money sink, and you're not going to see it for two years. But the issue is security: if you stay on really old stuff, eventually you could risk having a much worse incident that's a security-related incident because you didn't upgrade. So that stuff becomes harder. The conversations become multitudes of conversations that you're having with a lot of different people because you don't have a centralized function. So I think we're all figuring it out still. I do like your idea, by the way, of BUs owning different things. I think people are going in that direction. As painful as that is, because I say it's painful if you were in the other boat and now you're busting things up.
Bart Farrell: It is what it is. Oh, go ahead, sorry.
Daniele Polencic: Keep going, Bart.
Bart Farrell: We talked earlier about cost optimization efforts. It comes down to questions of responsibility, it comes down to best practices and enforcement, something that nobody likes, having to have lots of reminders that if we're deciding to do this, everybody needs to do this in a uniform way. And that's not isolated when it comes to platform work. In platform engineering, enforcement often means policy, standards, saying no to unsafe patterns to avoid risks. But the word itself can carry a lot of organizational and interpersonal weight. So why is the human side of enforcement as important as a technical mechanism, or perhaps given this conversation, sometimes even more important?
Molly Sheets: Why is it as important? One, you could lose money. We're talking about saving money, but you could just lose it, right? Because people are deprioritizing those objectives. And this is the constant battle of tech debt, right? And I don't like to use the word tech debt. I like to use the word quality. Stole that from Alan Page. And it's because if you hear about quality, you're forward planning on all this stuff. So you design a team, for example, to say, we are going to update this many times per year. This is going to be our release cadence on how we do maintenance versus maintenance being something that you are budgeting for every quarter, if that makes sense. I try not to use the word enforcement too, because people hate it. To your point, they hate that even though it's important. But I feel there are standards that you need alignment on: "We're not going to be on eight-year-old instance types. We're only going to be on up to three-year-old ones." It kind of depends on your savings plans and how you buy infrastructure and that sort of stuff. But having high level goals on that sort of stuff, even if you are in a split standard model, will at least enable teams to stay accountable to those things. I think you need both because you'll lose money. You'll lose in security. And you'll become really inefficient.
Bart Farrell: Sorry, go ahead, finish.
Molly Sheets: I think that's even more important now in the AI age. But people think AI is going to make it easier to do maintenance. Not if it's everybody doing maintenance for themselves, because they don't really understand. They're just going to take updates. People think they take updates and they'll just be safe. But that's not what I've seen. I've seen teams take updates and sometimes it blows up the whole system. So you have to have a strategy.
Daniele Polencic: Ooh, this is very interesting. Tell me more.
Molly Sheets: 15 minutes left. I am not going to throw specific vendors or open source libraries under the bus.
Daniele Polencic: No. But I'm interested in which way. I'm very much interested in these AI tools. And I do use them a lot. And I try to use them to do my job. In the past I had to do debugging myself, but now most of the time I just use AI to help me do the debugging. And what used to be a two-hour debugging session is now probably more like five minutes and me deciding what is going to be the next step. But you're saying that things are blowing up. And my interest at this point in time is less about the tech and more about people. Because most of the time when you're debugging, the first step is to analyze the problem and figure out what's going on. And the second step: understanding. And then the third step is to take a decision. But where is the blowing up happening? Is it a people problem?
Molly Sheets: No, it's updates. So we're all releasing faster with AI, right? And we think our releases are stable. And now we're going more towards environments where— we were doing this anyway— we want fast response time if something is unstable and to be able to either revert or get a change out the door. What I'm in engineers, that's great. And people are used to that. But what I'm worried about is we've got a lot of cross pollination of roles, which is awesome. People are learning new things. But the biggest thing we need to still educate on is: Don't just take updates. Test those updates in cell-based blast radiuses first in a strategic release way. Because I think people think: "If I take the update, I'm safe." I have definitely seen updates of networking infrastructure that were not stable, that we took, and we were testing them. And so I want people to know: Yay, awesome maintenance. Do it. AI makes it easier. But you need a good release pattern if you're going to do that. And be prepared to revert it.
Daniele Polencic: But it's also interesting because you're describing this. I just rolled back a few years and it was like, "Okay, I receive an update. I am not very happy because this means that next sprint or the next day is going to be me creating something or me trying to update something and crossing my fingers that everything is going to be fine when I deploy it." But right now with AI, what I do a lot is just create a temporary environment that probably takes me about five minutes because these agents can do the work on my behalf, then I do the test and then I tear it down. What used to take me days now can take an hour or so. I would expect more testing, but you're saying that actually people are trusting the AI more, skipping this process?
Molly Sheets: So we skipped a lot right? There were people who said: "I'm anti CI/CD," and now they're saying: "I'm pro agents," and they skipped the journey in the middle, which is creating cell-based releases. So canary is your biggest example: going from 1% to 20% to 25%. But another cell-based release could be: "We have a small customer pool that we're going to deploy this to. And now we have a bigger one. Now we have a regional customer pool." The EKS team at Amazon does this. They did a couple of re:Invent topics a few years ago about how they did cell-based release models. That is still, if not more important in this future than just testing it yourself because you can never truly mimic production. And so going back to the original topic, which is the multi-tenancy piece, if people are going to own their own stuff, and they're going to do less multi-tenancy, then you have to do cell-based, you have to figure out what your cell-based release strategy is for what you own now, which is everything that my teams were looking into in terms of how we structured things. One of the reasons to standardize is you can now say, this is what a release looks like across all these different environments. You don't release to your revenue generating workload first. You release to your Jenkins farm first. If it breaks there, then we can revert. I went off into the deep end in a lot of different ways on that one. But I hope that makes sense.
Daniele Polencic: It does. And I think to me, it's very interesting because we're touching on things, we're talking about multi-tenancy, but we cannot detach multi-tenancy from the tooling, from the people, from the business goals. Everything is somehow interconnected, right? And we are also in this transition phase where we've got these different workflows that are appearing. And not only that, but also some of us are having a crisis of identity. Others are embracing new ways of working. Someone is just merging PRs without reviewing them. The crazy things that are happening are just endless at this point. And as a profession, we're going through, I think, what is a little crisis, right? So it's interesting to see these patterns coming together. So to summarize what we discussed so far, I think we've got all of these different areas of multi-tenancy. I want to take something with me after this podcast, something I can take home, and I can tell my students when I teach Kubernetes. Is it a single multi-tenancy for all your clusters and don't use AI, is that what I should tell them?
Molly Sheets: No, they should use AI, but I don't know if you follow Charity Majors' work closely. She's kind of been putting out this idea of "What if we don't review PRs?" Think through the problem. Regardless of the philosophy of whether you should review PRs or not, or whether they're all AI written now and super verbose and it's annoying, what happens if it goes wrong? And I think that stuff was questions we had been asking for the last 20 years anyway. And a lot of tenancy design is based on what happens if it goes wrong. So if you do want a multi-tenant, okay, cool. What are all the pieces we need to have in place to do cell-based releases? So in the case of an application, you're talking about canary releases. I know a team that has different release namespaces. So they don't have namespaces as a hierarchical construct for the OU. They have release namespaces. Maybe if you're going to do load testing, you do it in a pool inside a bigger cluster instead of having a separate load test cluster because now we're managing another cluster. But you still have isolation in the pool. And I'm not even thinking about networking and IP and wrapping the network around the cluster. And you can still blow up the cluster with that, right? You load test too hard in a pool inside a larger cluster. So just thinking through the problems is still really important. And you can think through them with AI. That's awesome. AI can help you come up with these. But I think all that stuff is more relevant than ever. And because people are moving faster, they're going to move faster in the trust. They're trusting a lot. So spend some time to design the architecture in a way that you trust it if you break it. That was a long answer.
Bart Farrell: It was a good one. And I trust you enough, Molly, to answer this following question. Something that's come up time and time again in this conversation has been the importance of understanding that most people work in companies and those companies have objectives and they have business models. And it seems like there's a pretty big gap between people's understanding of the value that their job provides, what's actually necessary, what's a must, what's a good to have. We started out talking about for small companies, does it even make sense to have Kubernetes? And we get into more levels of complexity, enterprise level, all these different things where budgets get bigger, egos get bigger. But it seems like there's a fundamental misunderstanding of a lot of people about what their job is in terms of advancing a company's business goals. What's your advice for engineers out there that might be losing sight of that?
Molly Sheets: Wow, that's so tough because at the same time, your company— as an engineer that's growing— they also have to give you opportunities to explore new tooling. And there's all this new tooling now that's come out. And so I think what's hard is that even though we're moving really fast and doing everything, larger companies are big ships that are really hard to move. And if you really want to have a lot more control, go join a startup, go join a smaller company. They need that right now. But don't put it back on the larger business and say, you owe me this chance because the business is going to try. I think businesses are trying to grow engineers right now. At the same time, there's so much tooling and their business objective isn't necessarily to just do vendor tooling. It might be in the case of the industries I've been in, make games. The business objective is to make and release games. And so there isn't going to be all this opportunity to try 10 different tools necessarily. There may only be limited investment in certain areas. And so I think engineers have to remember where they work and what the business objective is and really question themselves: "Is the thing I'm playing with really going to help the business? Or is it something I just want to learn for fun?" If it is something you want to learn for fun, do it on your own time or go find a company that needs that. And that's OK. I'm in between jobs right now, so I can luckily say this with a very straight face. I have sometimes been in a place where I wasn't getting the opportunity to do the thing. But the business didn't owe it to me because that wasn't their business objective. And so, I want to do GPUs right now as part of it. And I also want to still build games on the side. Go find a company that's doing that. So, that's my thoughts. Oh, Jesus. We'll see.
Bart Farrell: We're looking at you, NVIDIA. We're counting on you. Read Dan's book. Sponsor the book.
Molly Sheets: Do that. I'm going to actually send this.
Bart Farrell: Sponsor our co-located event at KubeCon. Give us permission to use your logo and a long list of other things. Terms and conditions apply. Molly, you've already spoken about what you're doing right now, but anything else that's next for you, what can people expect from Molly?
Molly Sheets: I'm learning. I've never been one that's like: "I hop from a job right into another job." I always take some time to self-invest. And I have just some random advice for people that's unsolicited that they probably don't want: I save $300 a month for just my own education. And I saved up about $10,000 on that. And now I'm blowing it. I split it with my husband. And some people save more, but I just want to learn for a little bit here and make the right choice for the next thing. And the two things I'm investigating are: Should I run a games business again or do I go dive into GPU splitting, which is hard. And I did some interviews and I realized: (Can I cuss on your podcast?) "I do not know this shit." And I think it's OK to self-reflect and say: "Yep, that was the bar. I don't know this. I'm going to go learn this." So that's where I'm at.
Bart Farrell: That's going to be a great hook for the podcast, too. And Molly, if people want to get in touch with you to continue the conversation about the many different topics we touched on today, how should they do that?
Molly Sheets: As Bart has shown you all, I'm way too chatty on LinkedIn. It's my preferred platform. I'm not on X. I'm not on other places because that's just where I invest. So LinkedIn is the best way to get in touch with me. I think I respond on there a lot. I don't like to post my email publicly because people have abused it, which I'm sure you all understand.
Bart Farrell: Daniele does not have my email for that very reason. We exclusively communicate via Slack or occasionally in person in UTC plus eight where I work over next is happening. Dan, anything else to mention?
Daniele Polencic: No, I've got so many AI questions, but the time is running out. So I'm just going to shut up now.
Bart Farrell: We'll have to schedule our next episode. Molly, it was great having you back with us and look forward to seeing your next steps. Take care.
Molly Sheets: Thank you.
Daniele Polencic: Take care. Bye.
