How Kubernetes really works: Contributors, governance, and hidden complexity

Bart: So first things first, can you just tell us who you are, what you do, and where you work?

Note: While the transcript doesn't contain specific technical terms to hyperlink, I noticed the speaker works for the CNCF, which could be a relevant link for context.

Bob: My name is Bob Killen. I am known as Mr. Bobby Tables across various platforms. I currently work for the CNCF as a Senior Technical Program Manager. Before joining a year ago, I was at Google in their Open Source Program Office (OSPO), and I've been involved in open source and the Kubernetes cloud native space for quite some time.

Bart: And what was it about Kubernetes that first drew you in? How did that start?

Bob: Originally, this is a bit of a story. Jeff, Sika, Jeefy, and I were working at the University of Michigan Hospital. We were developing our own container platform to run clinical workloads, but that didn't work well. We moved to Docker, which worked better, but we were looking for more automation. Around 2014, we saw Kubernetes emerge and jumped on it very early—perhaps earlier than we should have. We were engaged in the project from the beginning, back when chat was on IRC in the Google containers chat room.

For our purposes, we temporarily left Kubernetes and switched to Mesos, which was more stable at the time. We re-engaged very heavily in 2016 and 2017, becoming much more active in the project during that period.

Bart: And this is an ecosystem that moves quickly. You've worn many hats. How do you personally stay up to date with all the different changes that are going on?

Bob: The best way to track Kubernetes developments is to find the enhancements tracking spreadsheet. Each release of Kubernetes gets a new GitHub project board that outlines what features are in development, their current state, and other relevant details. This allows you to quickly understand what's being built and its current status. Additionally, paying attention to the kubernetes-dev mailing list provides general Kubernetes updates.

Bart: And in terms of people that have helped you level up in your journey, is there anyone you'd like to mention or give a shout out to?

Bob: Two people specifically would be Paris Pittman, who was kind of my mentor in ContribX. She eventually stepped down, and I took her seat. George Castro was the one that really dragged myself and Jeefy into actually contributing to the project. Circa 2017, at KubeCon, I met George previously at our meetup. I had gone down there early and texted him, "Where are you?" He gave me directions and I went to the maintainer summit, the Kubernetes contributor summit. I felt I shouldn't be there, as I wasn't a real contributor. He handed me a T-shirt and told me to go sit in the room. That is how I pretty much became a contributor.

Bart: My start with being a contributor was through contributor experience and attending a meeting. This is not the same as having George throw a t-shirt on you and say, "Hey, go there and sit down." But I think there's something to be said for participating, listening, contributing, and giving other people spaces. After receiving the t-shirt, what were the first contributions you started making?

Bob: The first real thing I did was at the Kubernetes contributor summit, where I met Paris in person for the first time, as well as Spiff XP, Christoph Blecker, Ben the Elder, Tim Hockin, and a whole bunch of people. We started talking, and from there, I got more involved in ContribX because George was also heavily involved at the time. It was something I could engage with pretty easily as a hobby contributor without having to sink a lot of time into it.

Bart: Bart Farrell: Kubernetes means different things to different people. For you, what is Kubernetes?

Bob: Kubernetes itself is a substrate or underlying thing that holds up the new web and infrastructure these days. It's crazy just how far it's gone—you find it running in planes, cars, and edge devices. When it comes to how ubiquitous it is, there's the Linux kernel and Linux. And then there's Kubernetes. It's kind of crazy to think about where it is in the space.

Bart: From a technical perspective, you mentioned how ubiquitous Kubernetes has become. Now, I want to focus on the people side. Can you give us a bird's eye view of Kubernetes as an open source project with contributors from all over the world?

Bob: It is a great mixing pot. These days, it's a little bit harder to get involved just because it's a much more mature project. However, you will find a whole slew of passionate people who want to help, want to make sure it succeeds, and want to ensure it doesn't break anyone using it in planes, cars, and wherever it's running. For me, it's about people who really care about the space.

Bart: And how does the contribution actually travel from a KEP to something real that engineers run in production?

Bob: Before the Kubernetes Enhancement Proposal (KEP), which tends to be a large-scale feature, significant work is required beyond writing the proposal itself. This involves meeting with tech leads and experts in the specific code base area, getting their buy-in because they'll have to review the KEP and associated pull requests.

These days, it is difficult to get features into the core Kubernetes project. As a result, a lot of feature development is now happening out of tree in Kubernetes Special Interest Groups (SIGs). For example, the Gateway API and Kueue were developed in this manner. This approach allows for more rapid development outside of the core project.

Bart: And to go back to the people involved in this project, why do contributors show up? What motivates them? What do they get out of this? What's in it for them?

Bob: For everyone, contributing is different. Some people do it as part of their job, while others, especially students, see it as an opportunity. Contributing can look good on a resume and help build credibility. For me personally, the project is interesting, and I continue to keep tabs on it, even though we had to switch to Mesos back in the day. I find the patterns being developed fascinating.

I know it might sound like a cop-out, but everyone has their own reasons for getting involved. The project tends to be very sticky, and people who find their space often stay around.

Bart: People that have now been involved in this project for a decade don't necessarily last all that time. For those that are there, it's obviously for different reasons. You mentioned the backgrounds of some of the contributors: we have students, professionals, vendors, end users, people who are paid versus volunteers. What's the mix like? Who are these contributors?

Bob: These days, it's mostly professionals who work on Kubernetes either as part of their day job or adjacent to it. I know a couple of people who work on Cluster API providers that aren't technically part of their job, but they use it. When they encounter bugs, they have the know-how to contribute back.

We do have students and others who get involved to help with documentation. Program managers help with triage and other organizational skills. Most chair roles in the community project tend to lean more towards organization and people management.

The biggest way people can level up in this ecosystem is to be engaged and ask meaningful questions. Many people say, "Please help me contribute" or "Please teach me this," but they haven't done proper research. If you come to the table having looked at the codebase, a potential KEP (Kubernetes Enhancement Proposal), or specific issues, and say, "I'm interested in contributing to this feature. I've looked at the code and read some of its history. I have a few specific questions" - that shows you're engaged and comfortable doing the work yourself.

Maintainers are often strapped for time and can't handhold as much as they might want. Self-starters who can learn with just a little guidance will go far.

Bart: I heavily agree. When you see that someone's made the effort, they're comfortable being uncomfortable in environments where they might not know everything, but they're taking the time to read the docs and attend KubeCon meetings. You've got tons of talks from previous KubeCons you can watch, with lots of resources available. If you're really motivated and take the time to do it, I think it makes those conversations much more fruitful, and the questions will be much more targeted rather than a generic "help me contribute". Show me that you've done the homework first. I really like that.

Bob: I literally get 50 plus DMs a week asking, "Teach me." To be honest, I ignore them unless the person has shown more initiative in their messaging. I just don't have time to go through all of that.

Bart: There are ample resources. If you type "how to get started" in the CNCF contributor guides, there are lots of resources available. Now, we're talking about a large number of people involved in these projects and the infrastructure necessary to keep this going. Who organizes and funds all this, and what's the balance between vendors and end users?

Bob: It largely comes from the vendors in terms of both funding and people time. The larger vendors have also donated resources—Google, AWS, Azure have donated cloud credits so that Kubernetes can do its testing. For those who might not be super familiar, Kubernetes testing for every release involves spinning up around 5,000 nodes, which can get expensive very quickly. Overall testing costs are in the millions of dollars a year, and we wouldn't be able to do it without vendor support.

The CNCF itself provides top-level governance through the technical oversight committee and makes recommendations, but the products are mostly allowed to self-govern. The big thing the CNCF can do is help escalate concerns to a broader group. They facilitated credits from different vendors and put on events like KubeCon, Kubernetes Contributor Summits, and Maintainer Summits, providing opportunities for engagement.

I personally love to see more end users get involved in contributing directly. You can de-risk your own usage of the project by getting involved and help shape it by providing feedback. All CNCF projects are desperate for meaningful feedback. As an end user, you can be more engaged, and your feedback will be listened to. In various surveys, they don't get many responses, and you sometimes need to block out the loud voices asking why something doesn't run on a 10-year-old 32-bit Raspberry Pi.

Bart: Okay. And last but not least, what's something about the Kubernetes ecosystem that most people don't realize?

Bob: I think there are many components to Kubernetes. When people think of Kubernetes, they think of Kubernetes core. But if you look at Kubernetes itself, there are 400 GitHub repos. There are a lot of moving parts and interconnected things. This can go back to some of the invisible work, like documentation, testing, and contributor experience in trying to streamline the project itself and make it easier to contribute.

There is just a lot that goes into running a project of that size. These days, Kubernetes has around 1,800 organizational members and they've had something like 90,000 contributors. When it comes to thinking about a project of that size, it is like trying to run your own business or organization. It can become unwieldy and difficult to manage.

Bart: Perfect, Bob. That's all we need for today.