Immutable Linux for Kubernetes Nodes

Bart Farrell: Mauro, welcome to KubeFM. Can you tell us about who you are, what's your role, and where you work?

Mauro Morales: Thanks, Bart. Thanks for having me. Yes, my name is Mauro Morales. I'm originally from Guatemala, but I live in Belgium. I work for Spectro Cloud. I am part of a team called the Open Source Team, where we work on a project called Kairos, which is a CNCF sandbox project that helps you transform traditional Linux distributions into what we call immutable image-based distributions. I also co-host the Cloud Native Brussels chapter here.

Bart Farrell: Very good. And you mentioned Kairos. What are three emerging Kubernetes tools that you're keeping an eye on?

Mauro Morales: Yes. And the good that you mentioned is very much related about Kairos, the things that I'm touching all the time. I'm sure there are more interesting things that people would look for, but at least that affect the area where I'm working. The first one would be Spegel, if that's how it is pronounced. If not, I'm sorry. It's a peer-to-peer OCI distribution so that the images that you're distributing across your clusters don't need to be downloaded on every node, for example, but that they are strategically and smartly distributed across your nodes so that you consume less network, less bandwidth. I find this kind of tools good because they are efficient, but also they are more cost effective, right? So they are beneficial in my opinion. The other one would be Open Cluster Management. I think we are seeing more growth of people needing to put clusters outside of one unique cloud. So how do we manage these hybrid or multi-cloud setups for clusters, right? And having a framework to do that. Personally, I think it's better. Many of these problems are better if we use a framework that has already solved that problem instead of coming up with our own solution. This one is also, if I understand correctly, a CNCF incubated project or is becoming one. And the third, not one tool in particular, but I would be very interested to see more AI-assisted tools for Kubernetes coming out. This is because I myself work on the underlying layer of Kubernetes. So the more help I can have, for example, on how to approach certain problems that I don't see in my day to day, the better for me. It's also the AI bubble at the moment, right? So I'm just interested to see what kind of tools come out there. I've played a little bit with the K8sGPT, for example, and I've seen kubectl AI, those kind of things that I'm talking about, and I would like to see more of them.

Bart Farrell: Mauro, one of our podcast guests, Eric, asserted, if you want to learn infrastructure well, you need to know your operating system very well. What specific Linux areas do you find most relevant for Kubernetes work?

Mauro Morales: First of all, I pretty much agree with what Eric is saying. I think that whenever you're working on abstraction layer, ideally you are fluent in the underlying layer. It doesn't mean that you have to be an expert because otherwise it might be too much for us. But in terms of Kubernetes, you would benefit from knowing what Linux processes, knowing about networks, knowing what cgroups and namespaces are. All of these things have been abstracted from you in Kubernetes, which makes the everyday easier. If you know how they work, you can architect better, in my opinion, instead of just blindly guessing. It also helps you debug and it's kind of like a safety net, right? Like if you know how things are underneath, then if you don't find the answer at the Kubernetes layer, you start looking at the underneath one.

Bart Farrell: Now, our podcast guest, Mircea, argues that you can spend time securing a base operating system like Ubuntu, or adopt a minimal distribution such as Talos that offers a reduced attack surface. He hinted that the latter requires a lot less work. How do you secure your Kubernetes nodes? And is there an operating system you recommend running on Kubernetes nodes?

Mauro Morales: So let me just add a little bit here to what Mircea is saying. I think minimal is indeed very much wanted because the less attack surface indeed the better, but I would add that trust is also very important. And when I say trust, I mean which components are put together to build that operating system. In my opinion, it's very important that the components have been vetted by the community. These are open source projects that are being used by many projects out there. So that in terms of security and patches, a lot of people have put their eyes on it. A lot of people have tested it out in the field, right? So I think the ideal scenario is minimal plus trusted together. That would be kind of like the sweet spot. So how do we secure our Kubernetes nodes? Basically, from the Kairos eyes, the idea is to transform the traditional view that you are upgrading and mutating your systems and converting them into image-based and immutable systems. So all Kairos flavors of nodes are immutable. This reduces drift almost completely. And it also means that if someone is going to attack your node, they have much more limited options that they can do there because they cannot just come and install packages. It's just not possible, right? So that would be the base layer we could say. Now for users that really need a lot of security, let's think about someone that is deploying a Kubernetes cluster, physical machines in a grocery store or even out in the field, right? Like a farm, a desert, whatever. In those cases, you might want the highest level of security. And what we do there is called trusted boot, is a mix of using secure boot, using measurements through a TPM chip to evaluate that the initramfs, the kernel and other parts of the system are the ones that you specifically set you're going to be booting in that machine and also encrypting user data. All of that together really gives you a system that is almost unhackable. And I would say almost because of course there's always someone out there that knows how to, knows better than everyone else, right?

Bart Farrell: Present company included. Mircea argued that you shouldn't use a Linux distribution like Ubuntu to run your Kubernetes nodes in control plane. If your goal is to host Kubernetes, you should consider minimal distributions like Talos or Bottlerocket. What's your advice on operating systems in Kubernetes?

Mauro Morales: Well, here I would say if you are allowed to choose any Linux distribution, then I would probably agree that you should go into what we call a special purpose operating system that is designed for running Kubernetes workloads. But the truth is that a lot of organizations don't have that. For example, if you're working in a bank or a highly regulated environment, you basically have to take what your security team is telling you that you're allowed to take. Or if you're running certain workloads on a specific device, let's say NVIDIA for AI nowadays, you also need to run a specific Linux distribution at the moment. That's why, at least on the eyes of our project Kairos, we think that users should bring their own image. That's why what we do is we transform that image that they are bringing into what becomes a special purpose operating system for running Kubernetes. That being said, I do agree that the smaller the surface that you can have, the better. Therefore, we also have our own Linux distribution called Hadron, where you will get kind of like the best of all, right? If you are able to make that choice, of course.

Bart Farrell: So Mauro, what's next for you?

Mauro Morales: I think there's a lot of Kairos and Hadron still coming, and I'm excited about that. It's funny because we've been doing Linux for so long, but more Linux is happening. So I'm looking forward to that. In terms of the technical parts and also the community. So I'm trying to go out and present in different conferences. In March, I'm going to be at KubeCon. In May, I'm going to be at Cloud Native Days Italy, again, sharing about Kairos and in general about operating systems and Kubernetes. I also think we're going to be learning probably week by week about AI tooling and whatever. So I'm very excited about that. And on the personal level, learning how to be a dad, running a little bit more, and learning some Dutch.

Bart Farrell: Very good. Well, it's quite busy. You've got a lot of plans. And you mentioned the stuff around AI and tooling, but looking at Kubernetes turned 10 years old a couple of years ago, looking at the next 10 years of Kubernetes. What kind of things can we expect, both AI and non-AI related? What do you think?

Mauro Morales: Very good question. I think the next 10 years are going to be boring in the good sense of boring, right? I think we've seen now that things are stabilizing in certain ways. So more security focus, which is what happens when things get boring, right? More compliance, more standardization. On the other side, on the AI part, I think, like I was saying, more tools are going to come, a lot of them probably, because we are seeing projects that are using Kubernetes to deploy Kubernetes. It's getting quite complex. And I think the more tooling that we can use through AI to understand that complexity, the better.

Bart Farrell: And Mauro, if people want to get in touch with you, what's the best way to do that?

Mauro Morales: The two best ways are on my LinkedIn page and on the CNCF Slack. Both of them you can find in my website mauromorales.com