Kubernetes v1.28: Planternetes

Bart: Welcome to KubeFM, a podcast to expand your Kubernetes knowledge, follow the latest tools and trends, and learn from the experts. My name is Bart Farrell and I will be your host. I'm a CNCF Ambassador in Content Creation. In today's first episode, we are talking to Grace Nguyen, who led the release team for 1.28. She's got a lot of experience on the release team, first starting out as a shadow, moving her way all the way up to release lead. What's it like being a release lead? How many hours do you have to spend per week? What are the necessary tools and tips and tricks she's learned while passing on to the next folks that will be taking over in this position? Check it out. Welcome to the first episode of KubeFM. My name is Bart Farrell. I'm a CNCF ambassador and content creator. And I'm joined today by Dan, if you'd like to introduce yourself.

Dan: Yes, I'm Dan. I'm one of the instructors at Learnk8s.

Bart: Very good. And we are also joined today by the 1.28 release team lead, Grace. Grace, how are you doing?

Grace: I am great. Thank you for having me.

Bart: Fantastic. Great to have you with us. So Grace, just to get a little bit of background, how do you get into Cloud Native? What was it that brought you to this ecosystem?

Grace: Yeah, so fun fact for having finished university yet, I am still finishing my fourth year here at the University of Waterloo. And so part of the Waterloo system is you have to do six co-ops or internships before you graduate. And so part of that process, I got into DevOps and then Kubernetes. And then during the pandemic, I got a scholarship to attend KubeCon virtually. I got to learn a little bit about the release team and how it's been around since. And it's been over two years.

Bart: So you've seen quite a few different releases. What was the process of getting into the release team, the kind of stuff that you did in the beginning, as opposed to what you've been doing more recently leading the release team?

Grace: Yeah, so the SIG-release has this amazing program called the Release Team and anyone can apply to be part of it. And so we have this structure in place where you can be a shadow and then you can lead a sub-team. So that can be Enhancement, CI Signal, and then if you're interested, you can become a lead shadow and eventually lead, which is what I just did. So in the beginning, I think the hardest part about the release team was to get in. It's quite competitive these days, and we're working on that. But in the beginning, I was quite intimidated because I use Kubernetes in my day-to-day work, but I had no exposure to open sources at all. And so it's been quite a ramp up.

Bart: Regarding that ramp up process, once upon a time, Dan and I were in university, but it was another decade. But with that in mind which the skillset in mind that you had, from being in university, like you said, ramping up, what was that process like? How did you level up? Or some of the resources or places that you went to increase your skills?

Grace: Yeah, I think with ramping up with the community, it's an interesting process. I think a lot of it was going to KubeCon and meet people at the contributor summit and ask people questions, and that gave me a really good overview of what people's roles are in the ecosystem. And then as part of my role as lead, I was able to pull people in quickly because I understand who did what.

Bart: Very good, like you said that who's who is crucial. It was also instrumental in my process getting on board in the CNCF. So big shout out to all the folks in the Contributor Experience Upstream Marketing Group that helped me get, you can say, it's sort of my foot in the door and being able to figure out who is who and what's going on. It's a very large ecosystem. That being said, let's get more into this particular release. All right, so Planternetis, the logo design, what was the process behind that? Did you have any input? What does it mean to you?

Grace: Yeah, so unfortunately I have full creative control. So I didn't have much input. But it is summer here in the northern hemisphere and I'm a big houseplant fanatic. And so I thought it would make a really good symbol for the release team because you know we're so diverse. We literally come from all over the world and we have different sets of background, but each of us plays a really critical role in maintaining the ecosystem.

Dan: Did you draw it?

Grace: I did draw it myself. Yeah, I'm perfect. Oh, wow.

Bart: Have you had you drawn logos before?

Grace: I have not, I figure now's a good time.

Bart: Okay, good. So we got some custom swag probably on the horizon. That'll be coming out. Good, with that being said, you know, a lot of things happened in this release. And as you said, you know, wide and diverse range of folks participating. We got 45 enhancements, 19 entering alpha, 14 have graduated to beta, 12 have graduated to stable. We've got 911 companies and about 1,440 contributors in terms of individuals. Are there any, you know, you have been in previous release cycles. Were there any, you know, companies or individuals whose contributions that really stood out that you'd like to mention that really caught your attention?

Grace: Not one, anyone in particular, but I would say, you know, folks who show up and comment on a release blog and they're not part of the release team, I think really brightened my day. And so they just kind of show up and read the draft and leave some bits of feedback. That was tremendously helpful. But one group I would shout out to is the folks at SIG-Docs. Communicating is such a big part of releasing. And so the folks over there were so helpful with us, you know, helping us to get blocks out and then the feature blocks out and then the documentation that is actually on the website. So huge shout out to them.

Bart: Yep, can never, you know, over appreciate the contributions that are made there as sometimes somewhat behind the scenes, but nevertheless, very, very helpful. In terms of the enhancements that got through in this release cycle, which one are you most excited about? And perhaps which one are you least excited about?

Grace: I don't have a least excited about, but I am like folks, excited about the sidecar container one. So you can dive into that one first.

Bart: Perfect.

Grace: Yeah, so starting in 1.28, we are introducing the concept of sidecar containers. In fact, it's just another flavor of the init container. So it's the init container with the field restart policy always. And what this means is that this container will live throughout the life cycle of the pod instead of dying before the pod comes up. And there's quite a few implications for this. So the folks at Service Mesh are excited about this. This is good for, you know, updating your secrets. and getting them pulled into the pod. And then one thing that I have my eye on is just logging. You know, if this cycle container is going to be alive and die after your pod, that's going to give you valuable information.

Dan: Yeah, see, I think this is one of the most exciting for sure. I was just a little bit surprised. So the first time I read the KEP, I think what really struck me was, okay, so there is no like sidecar field. It's not an unique container. I mean, at the beginning I thought it was going to be another container. That was my expectation. But then reading, reading the documentation, then I figured out that it was part of init containers. And that was like very surprising. What was the reason for that?

Grace: So SIG, the release team doesn't dive too much into the... We leave full control to the SIG to make up their architecture decisions. So I actually don't know.

Dan: Okay. I think I read, you know, the sort of reason I read so far is that they didn't want to change the structure too much. So the best way to keep compatibility was to add this extra field. But I was just that, you know, sort of curious on We got these very different container types and it's so interesting to see how big decisions such as this one, if I collect it down and sometimes we've got our hands tied and we can't really change things too much. Okay, but that, yeah, that was interesting.

Bart: And I will apologize in advance for the very terrible joke, but when I did read about the recovery, generally available, you know, GA recovery from non-graceful node shutdown, your name happens to be Grace, doing things gracefully. Can you tell us about that? You know, talking about, you know, stateful Seth Ponds being stuck in terminating, graduating to stable. What was the process behind that? Anything you'd like to touch on there?

Grace: Yeah, so this was in beta in 1.26, so it had about two cycles to kind of sit and soak and get feedback. So what happens there is in scenarios of certain nodes shutdown, like if the node runs window, or if there's broken hardware or failure in the OS, this is essentially a mechanism for the pods in a StatefulSet stuck in that failure node to get feedback. error list it out into a healthy node.

Dan: I had a quick Googling because I'd read the description and it was like, okay, well, what is this? Why is the issue? What is the issue? And at the Googling, there's basically a gazillion pages of Stack Overflow saying pod stack pending, pod stack terminating, and what to do? And then, yeah, I think, you know, and then that's basically what I connected. I was like, oh, okay, this is the reason. And I think this is one of the most, you know, one of the most interesting features, seeing a problem which is so widespread being finally fixed. I mean, it took a few releases, but we have it. We have it with this release. Yeah, I was just wondering, one of the things that I was looking at when I researched it, was like, how do they do it now? I'd really solve the problem of removing these terminating pods when they're stuck before 1.28. And so, and I find, I sort of didn't find all of the answers. I sort of find people just forced to terminate it, pause, but it would be really interesting to see if there is any feedback from the SIGs on, you know, the operators doing it or any companies sort of diving into the problem and offering their own solution before you actually, you know, you use 1Centigrade.

Grace: Yeah, so I'm not exactly sure about the mechanism of how to do, but I know one of the goals of this feature is to ask or require user intervention when the note shutdown is detected.

Dan: That's pretty cool. Yeah, interesting.

Bart: Another thing that came up, validating mission policy with major changes with related to reason and field path. Is there anything you'd like to add on to that, talking about matching conditions for webhooks, things that folks should be keeping in mind?

Grace: So this feature is one of the ones that uses common expression language. I've only heard of this release, so I thought that was pretty cool. And there's another couple, like Webhook Admission Policy, that requires this as well. So I thought that one, the language is interesting. It is now in beta, still has to be opted in. But one of the cool things about Kubernetes feature is that if you have any feedback and the feature is still in alpha or beta, show up to the KEP and give us your feedback. That's the reason why there's one or two releases between the stages.

Dan: Okay, okay, I understand. Thank you for that.

Bart: Good. One thing I wanted to add though, particularly on the aspect of feedback, is like you said, it's quite a competitive process to get into these positions. You spent a lot of time, you know, in previous release teams in different roles. But that being said, there are lots of other ways to get involved. And one of them precisely being feedback. Is this something you feel that is still, folks maybe aren't taking advantage of it as much as they could? Is it something you'd like to see grow out more in the future? Are there any changes that are happening there to make that feedback more visible? Anything regarding that?

Grace: Yeah, I think it's definitely something I wish I seen more. So one of the features that's released, the moving pkgs.k8s.io to community infrastructure and something you can opt in, that is not something that I've seen a lot of. precisely a Kubernetes feature. It is kind of a meta feature, right? Because it's about how you get your package and how it gets distributed. And that feature is in alpha, although it doesn't have a very clear graduation criteria, because it's a community features. And so the precise reason it is in alpha is because we want folks to be able to, to give us feedback of what that experience is like using our community infrastructure versus using Google hosted infrastructure right now.

Dan: And how does the feedback come to you? Is it just like for PR? So do actually people participate in the mailing list? What does it look like? What does it look like? Yeah,

Grace: so there's just comments on the PR. So they would go directly to the enhancement author.

Dan: So there is no like, you know, one of these annoying pop-ups that you see or you visit the website and say, hey, what do you think about this feature? Ranging from one to five stars.

Grace: That would be funny. And that'd be like a dark pattern on Earth.

Bart: But I think nevertheless, the point is that it's, this is a transparent process in which there are lots of different ways to be involved. So for folks that might be applying for different roles in the release team, not necessarily getting that directly, doesn't mean that you can't be involved. And so I think that's a, it's a strong, it's a strong thing to keep in mind. Now we, we get to the, the part, you know, that, that can get a little bit, a little bit. I don't want to say controversial, but you know, talking about deprecations and removals. So first the removal of CSI migration for GCE PD. Anything you'd like to comment on that particular?

Grace: I want to comment on all the deprecations or removal as a whole. So we got like some CES plugin removal and the removal of CSI migration. I think these are relatively minor compared to the previous release. So usually we have a mid-release deprecation or removal block, which we do not have this time. So, The context I've gotten there is that a lot of the storage plugins from companies or vendors that were in Tree is now moving out of Tree. And so it should not be disruptive.

Dan: Yeah, I believe it's been a very long process and we are still paying for it. It's time and monotonous time to just migrate all of them to CSI. So I think I think we should be towards the end, I believe. Isn't it? Yeah, yeah. As in we migrated most of the entries, most of the entries, plugins are migrated. That's my impression, yeah. Yeah, I suspect very much, yeah. Very much overdue.

Bart: And with that in mind, I mean, having been in this role, seeing the whole thing, we're talking about a period of months, What for you were the biggest challenges or things that perhaps you didn't expect necessarily having been on the team previously, having this particular role? Were there any moments or tasks that you were doing that you found particularly challenging?

Grace: I think the sheer volume of everything is quite overwhelming. So for example, I was part of editing the major theme blog post, and I don't know all the things about all the major themes. And I honestly, I'm not sure if lots of people do. And so it's a lot of onboarding technical materials. The other thing is the sheer size of the number of features that we get. I think at Enhancement Freeze, which is the very first milestone of the release, we got 100 features, and then we get to 45 after code freeze. And so you can imagine the team that manages 100 enhancements coming in, and then at code freeze, kind of trying to match them with Kubernetes, Kubernetes PR, the code PR, and trying to make sure that there's no floating PR around. We don't have a good mechanism to do that right now, and so that worries me a little bit, but we are growing a lot. And so as part of the process.

Dan: Grace, what's your secret sauce? How do you, what do you use to keep everything in check? How do you organize your time? Would you have any special sauce or software you use? I mean, it sounds like a lot.

Bart: Like an immense amount of things to be tracking at the same time. Yes, what's your secret? Are you sponsored by Trello or how does this work?

Grace: Yeah, so before we were on a Google Sheet, and that was a lot of pain. So when I joined the team, I was on a Google Sheet. And so it was kind of scary too, like you would add something and you would be scared that it will break that macro over there on the docs team. And then we had migration over to the GitHub where that tremendously supported our process because it gave us a really good visualization of like, who's the owner? Because you know, just to fill in some of their fields. And then if things are open, pretty pretty visual as well and also helps with like filtering like it it's pretty cool Features are removed from Mouse so we can filter for all that good stuff.

Dan: So just to give an idea, I'm really curious, how much time a day is that? It sounds like more than a full-time job to me. Is that correct?

Grace: this is, honestly, this is a lot of work to the enhancements team. Huge shout out to them for managing this huge load. My day-to-day is to checking in to see how they're doing and if they need help. Okay.

Dan: Okay, understood. Wow. Okay. Sounds very, very intense. Yeah, well done.

Bart: And if was there any moment in particular that that that stands out where, you know, team coming together in a way that they didn't necessarily expect to? Any surprises along the way?

Grace: Yeah, surprises along the way. So we released on Tuesday and the Wednesday before release, someone DM me and we're like, Hey, Grace, like I saw this XXL Code PR merge, but I couldn't see a tracking thing for it in the forum. I was like, oh no. And, and, you know, it did, it was not opted in for the, for the release. And so the code should not be in the release. And so we have to do a rollback of that. And I think it's just one of those things that like. is almost certain every release that it's going to happen. I wish that it doesn't, but from the lore that people tell me, there's something that's always going to break. You know, either all your tests could go down on the Sunday of, or GitHub went down the day we released. There's an outage in the webhost admission, or webhost. Yeah, so there's always little things like that. I think one of the moments that really highlights it for me is the major theme blog. Because it covers such a wide range of topic, we really encourage the release team, but also folks in the community who just hopped in and, you know, gave it a read and gave us some feedback.

Dan: So would you do it again?

Grace: I am really tired now. Speaking of unsweet. The bottom line, every release lead is an emeritus advisor who has done this role before. And for me, this release is Leo. And so he has been tremendously helpful. And so I would consider him my backup as an emeritus advisor.

Bart: I mean, with that in mind, we previously talked about leveling up and ramping up, you know, Kubernetes knowledge. At the same time, the release team, I think, requires, you know, very specific things for folks that are out there who want to get involved. What are the, because like you said, it's quite competitive. What are some of the skills or best practices they should be keeping in mind if that's something they have on their radar? If it's an experience they'd like to have, what advice would you give?

Grace: Yeah, so actually our 1.29 Shadows application should open soon, by the end of the month. And I really, really encourage everyone to apply. When I applied, you know, I was a first year university student. I didn't have any open source experience. I think showcasing your open source experience, no matter how small, you know, whether it's just you had a docs fix here and there, shows us that, you know, you're willing to show up and commit. The other thing is like most of the folks are unfortunately going to get rejected and I encourage them to apply again. It is something that we do look at.

Bart: Like you said, there is no wrong way to approach it. There's no matter, like you said, no matter how basic your experience might seem, just about cultivating that and understanding that it's not just going to be a one-time thing, there are, you know, there are continuous release cycles. So there are different opportunities to get involved. And like you said, it touching on so many other SIGs that are in the ecosystem that are also going to be a part of that. It can be approached from many angles. So I would, I would say to, you know, folks don't get discouraged. And there are, and once again, don't get discouraged. opportunities to provide feedback, as we said earlier. So there are many, many ways to be involved. You already gave a shout out to Leo. Is there anybody else from the team, from the previous teams that you would like to give some credit to, some recognition that helped you get to the position where you're at today?

Grace: Honestly, I think the entire team, you know, doing their precise job is the reason that I'm here today and being able to speak about this. You know, we play, like each of us play such a critical role in the ecosystem. But I do want to congratulate the incoming lead, Priyanka, who works with me for multiple releases in the Enhancement team. And she was also part of that, you know, migration from the Google Sheets to the GitHub force. So she has such a good understanding of the release and she's so well equipped to run the next release. And then Xander, who was our previously one of our release lead in 1.27, he's coming back as the Emeritus Advisor. And so I wish him the best.

Bart: Very, very good. Like you said, there are a lot of folks that have been involved. Just thinking about the migration from Google Chats to GitHub makes my head tremble. So I. From a distance, appreciate your work very much and patience and understanding and making that happen, you know, change management as much as we're talking about technology, a lot of this is about how do we get folks on board and committed to something that we know is going to have some bumps in the road, what we believe is better for the future release teams. You will be speaking, I believe in a webinar in a couple of weeks about this with the CNCF, is that correct?

Grace: Yes, it is on August 6th we issue a PR to fix the new date app. But yes, it should be on the CNCF. We will discuss more of the features.

Bart: Dan, any final questions?

Dan: Yeah, I've got one final question. I just like, you know, and whatever the reason you release, I go on the website, I go on the blog and just read, you know, what's new? And then every time I just like. I basically realize I know nothing. I sort of understand how things work, but in reality, it's a reality check for me. I basically just face the reality that I know very little. And the question for me is like, sometimes, and if you're a release lead, I believe you touch on so many subjects, so many topics, so many things. And if I were to do that job, I would be so scared to one, mess up things, two, look stupid because I don't know. what half of the features are. How do you do it? How did you do it? I mean, it's just like, it sounds to me, it sounds like an overwhelming job. So what was your secret to tackle this, your mindset to tackle all of these challenges?

Grace: Yeah, I think one of the best things, and I've been able, been better at this throughout the release too, is being able to ask for help. Because you write down, I think very few people in the Kubernetes realm deeply understand all the features that we put out in the major themes. And so being able to know who to ask as a community is a big thing for me. And also I feel like messing up is inevitable, unfortunately. I'm spoken to almost all of the previous release leads and they always tell me there's always that one gotcha. And so it's... making pins with what I've...

Dan: So I guess it takes a village to ship Kubernetes.

Grace: 100% oh my gosh we have what more than a thousand contributors and 900 comp. That's a crazy number.

Bart: It's ace. It really is. And it says a lot that, you know, that people see the results coming from this and they believe in the process. And because there are amazing folks such as yourself and then the people that you mentioned that are making it happen, but it really is the true nature of a collaborative effort. And which is why going back to Dan's question about how many hours are in Grace's day? What is, how do you bend, you know, space time?

Dan: She isn't telling the truth, isn't she?

Bart: because it's very, very impressive to tackle that. And while having all the other responsibilities that you have going on. That being said, what are your next steps? What can we expect from Grace in the future?

Grace: Yeah, I'm gonna sleep for the next two weeks because I finished my finals and the release. So really happy about that. Yeah, I also work in security. I support their self-assessment subgroups. So, you know, subgroups within Kubernetes come to us and we help them perform a security review and that part is quite fun for me. So, one has been done already and that's for cluster API. There's a report out on that. Currently we are working through helping the folks at vSphere CSI driver evaluating clusters. their stack. And so that's also a very cool process to be part of as well. So SIG-security folks are really friendly. If folks want to show up and look around and you know, that's another way to get involved.

Bart: All right. Well, looking forward to seeing whatever you do. You will be very public no matter what. And looking forward to that webinar with the CNCF to see the other folks from the team, the questions that might be asked. Dan, is there anything you'd like to mention before we wrap it up? Yeah, I think we're good.

Dan: And that's all. I just want to thank you, Grace, for coming and sharing all of this knowledge with us.

Grace: All right. Thank you for having me, you two.

Bart: Pleasure. Cheers. You're welcome.