Exploring multi-tenancy for my Kubernetes learning platform

Bart: In this episode of KubeFM, I had a chance to speak to Stefan Roman about the intricacies of building multi-tenancy in Kubernetes based on his work for a Kubernetes-based learning platform called Labs4Grabs. This episode will cover how to evaluate the differences between namespace-based and control plane multi-tenancy, the role of tools like vCluster and Capsule, and the trade-offs these options present in terms of isolation, scalability, and operational efficiency. Stefan shares key insights into how to approach multi-tenant architectures in a way that balances security, performance, and maintainability. If you're looking to sharpen your understanding of multi-tenancy strategies, this conversation is valuable because Stefan is also a boxing fan, which contributed to making this an exceptional conversation. This episode is sponsored by Kusari. Modern software supply chains are full of vulnerabilities, with limited visibility, and no one sees the full picture. Incidents are inevitable, but panic is optional. With Kusari, you'll know exactly what's in your infrastructure. The Kusari platform stores, tracks, and analyzes SBOMs and delivers insights. Securing your software chain starts with transparency, and transparency starts with Kusari. Now, let's check out the episode with Stefan. Stefan, can you tell me what three emerging Kubernetes tools you're keeping an eye on?

Stefan: I'm paying attention to Kubernetes itself because it's the most important thing in the Kubernetes world. There have been some great developments recently, particularly with Kubernetes 1.30, which includes new features for Persistent Volumes (PVCs) that I would like to use for Labs4Grabs. I've been searching GitHub and release notes, as well as reading what my colleagues and other bloggers have to say. I'm also keeping an eye on Kubernetes because it's a big part of Labs4Grabs. The GitHub release notes are great, and many people are contributing to the project. Additionally, I'm interested in the observability stack, including tools like Grafana, Prometheus, Loki, and Tempo. This is because, at my job, mishaps in configuration have caused significant cost spikes, such as unexpected API calls to S3 buckets. Monitoring these tools can help avoid such mishaps.

Bart: Very good. Now, tell me more.

Stefan: What do you do?

Bart: Where do you work?

Stefan: I work at a company called Labyrinth Labs as a DevOps engineer, and we specialize in designing and building cloud-native infrastructures, mainly in AWS. We have some Google Cloud customers, but this is 99.9% AWS, and we heavily focus on Kubernetes and serverless technologies.

Bart: What did you do before [cloud-native](What is meant by cloud-native in this context? Is it referring to cloud-native technologies or cloud-native applications?)?

Stefan: I was never an IT guy, although people thought I was because I knew what the Windows control panel was. My first job was in a data center, where I worked as an apprentice. This was definitely not a cloud-native environment, as I spent my time playing around with physical Cisco devices and configuring them. I also performed manual labor, which is rare in IT jobs. One interesting thing about data centers is that they have cold and hot aisles. The hot aisles are where the servers and networking devices output their heat, while the cold aisles are where they draw in cold air, and the air conditioning functions. If you have a lot of cable management at the back of the data center, in the hot aisle, you can get very sweaty, and then you have to connect something at the front of the aisle, where it's cold. Getting cold was quite frequent in the data center where I worked.

Bart: So, when you weren't getting hot and cold, you did eventually get into cloud native technologies. What was that process like?

Stefan: I never thought about getting into cloud native. I didn't actually know what it meant, as cloud native encompasses a large scope of things, technologies, and principles. I started working with AWS cloud, which was my main focus. I did some certifications, including the associate AWS certifications, and I really wanted to work on AWS cloud, using tools like auto-scaling, load balancers, and S3. I tried to find employers that used this stack, and fortunately, I found some that also worked with containers, Kubernetes, and big auto-scaled environments. It kind of just happened for me; I never actually wanted to go into cloud native, although it's great. After a couple of employments, I started to work with Kubernetes because one of my employers, Piano, had a paywall for online newspapers. Every time a huge event occurred, the auto-scale environment would spiral out of control, resulting in 200 servers, and when nothing was happening, there would be only two or three. They wanted to move to Kubernetes to auto-scale applications and cut costs. That's when I noticed Kubernetes as a key cloud native technology. It was quite new at the time, and I'm not sure what version it was, but since then, I've become a cloud native engineer.

Bart: Blogs and different resources help to stay on track. The Kubernetes cloud native ecosystem moves very quickly. What resources are your go-to places to learn or to level up?

Stefan: Reddit is a great resource, with r/kubernetes and r/DevOps being excellent communities. People post all sorts of problems, some easily solvable, some not as easily solvable. It's a great place to learn. There's a lot of marketing happening in those subreddits, so you can keep an eye on emerging technologies. Also, there are plenty of new technologies advertised in those subreddits. Medium is also a great platform for new technologies and all sorts of problems that people have in infrastructure.

Bart: If you could go back and give one career tip to your younger self, what would it be?

Stefan: I thought about this and decided to slow down. IT is a complex field with various machinery and moving parts. I would like to slow down and understand why things are failing. This is definitely good advice. For people who are starting, it's essential to understand why things are failing, rather than simply pushing aside failures and trying something else, as I used to do.

Bart: As part of our monthly content discovery, we found an article titled Exploring Multi-Tenancy Solutions for My Kubernetes Learning Platform. Let's take a closer look. To start, can you provide an overview of Labs4Grabs and explain the platform and the problem it's trying to solve?

Stefan: When I did my certifications for CKA and CKAD, I never did CKS, which is something I want to do. When I looked at the resources available online, I found great resources for learning, including KodeKloud and some Udemy courses. There are plenty of clever people who can pace the content in a way that is digestible, so you can learn as quickly as possible. However, I thought that creating a namespace or a pod, or creating a deployment, is not really something that happens in the real world. In reality, you have Helm charts that can do all this heavy lifting for you. The real problems arise when you have issues with Helm charts, templating, Kubernetes, kubelet, resources, hardware, and other similar challenges. I always fantasized about a platform that can teach people these kind of real skills. Labs4Grabs is kind of that fantasy - a Kubernetes learning platform that gives you root access to the environment. It's a Linux environment with a Kubernetes cluster that's usually up and running, but sometimes it's broken and you have to fix it. The platform definitely focuses on real-world problems, some of which I experienced in my career.

Bart: It's also nice that it's free. I imagine there are significant technical hurdles when designing a platform like this, such as KubeVirt, vCluster, or Capsule. What were the main challenges encountered during the development process, for example with NetworkPolicy, RBAC, or other Kubernetes features?

Stefan: So, definitely, security was a main focus. My fantasy was to provide root access to users and full access to the Kubernetes cluster, allowing them to run privileged workloads. Security was a major hurdle, particularly with multi-tenancy, as there are various solutions and technologies that approach this concept in different ways. I needed maximum isolation, and there are a couple of technologies that solve this problem, such as Kata Containers, Firecracker, and KubeVirt. However, learning these technologies and understanding how they operate was a significant challenge. I was concerned about misconfiguring something, which could lead to a security incident, such as someone gaining unauthorized access to the root cluster or reading sensitive information. As I lacked experience with these technologies, I did not trust myself, and this was a major issue. Another significant challenge was resource allocation, which could be managed using RBAC and NetworkPolicy. I had to limit the amount of resources to a feasible level, as I could not allow students to have 256 gigabytes of memory per lab. My goal was to find a balance between providing a good experience for the students and ensuring that the system could handle the workload without crashing or scheduling issues.

Bart: Now, many developers might assume that Kubernetes namespaces provide sufficient isolation. Why didn't namespaces meet your requirements for this learning platform?

Stefan: Developers might assume that namespace isolation is the solution. However, the problem with namespace isolation is that it is usually implemented when providing resources to people you trust. Since you cannot trust the public, namespaces may not be the best option. Namespaces are great technologies, and I researched a technology called Capsule, which addresses some of these issues. The main problem with namespaces is the significant amount of configuration required for namespace multi-tenancy, including RBAC. You need to generate a kube config, create a user, and define RBAC per lab. Additionally, you cannot run privileged workloads because running a privileged image allows users to log into your root cluster and access the file system or other pods. This makes it complicated and not as secure as one would think. For people you trust, such as in a development cluster, assigning a namespace to a developer can work well, allowing them to develop, destroy, and rebuild their projects. However, for public use, it is definitely dangerous.

Bart: Given the limitations of namespaces, you must have explored other options. What technologies did you initially consider for addressing these multi-tenancy challenges?

Stefan: Initially, I found vCluster and tried to wrap my idea of the lab and the content around it. vCluster is self-healing, so if you break the Kubernetes API, the whole vCluster falls over and rebuilds itself. The content I wanted to create involved breaking the API server, scheduler, kubelet, and other Kubernetes components, but this was not feasible with vCluster because it requires SSH access and is just a container that you log into via a kube config. A lot of the content I had planned fell apart because of this limitation. I still tried to use vCluster and adapt my ideas to its concept, but it ultimately didn't work out. The main issue was that vCluster runs a container, and if I wanted to teach students how to run privileged workloads, I couldn't do that because vCluster's fake Kubernetes API server schedules everything onto the host cluster. If a student schedules a privileged pod, they can exec into that pod and gain access to my cluster, which is not acceptable. Although vCluster has flags to prevent this, I wanted to provide privileged content. I also explored Kata Containers and Firecracker, but at the time, Firecracker required additional configuration to run in a Kubernetes environment, involving low-level setup like udev. I found a script to work around the problem, but I didn't feel the technology was suitable for public use. However, I've heard that AWS Lambda runs on Firecracker, so it's definitely being used in production. I'd like to re-explore Firecracker to see if it's improved, as it would be a nice solution.

Bart: Now, I know you mentioned vCluster. Can you elaborate on why it seemed promising initially and what your experience was like in terms of implementation, specifically on the syncer feature?

Stefan: So, vCluster is great technology. It's a fake Kubernetes API server that you deploy using a Helm chart, which creates a pod in your system within a specified namespace. This pod acts as a fake Kubernetes API server, and from then on, you receive a kube config file, which is a public Kubernetes configuration file. You can use this to log into your new Kubernetes cluster. However, vCluster doesn't have components such as a scheduler; instead, it uses the scheduler on your host Kubernetes cluster. This makes it a simplified version of a full-blown Kubernetes cluster, as it relies on several components from the base cluster.

When it comes to the Syncer feature, it has been great, especially for networking content. Many real-world applications communicate with the outside world and are accessed from outside, such as websites and API servers. The Syncer feature allows you to schedule a pod using the received kubeconfig. Although the pod is scheduled on the host operating system, it is initially inaccessible. By deploying a NodePort service, the service gets replicated onto your host cluster, making it possible to expose the service scheduled on the vCluster from the outside. You can deploy a website and access it from the outside via NodePort if you have the public IP address. Alternatively, you can deploy an Ingress object, but this requires some /etc/hosts configuration to access it.

The Syncer feature was excellent for networking content, and I felt sad to move away from vCluster due to its disadvantages. In a blog post, I mentioned that I would try to reimplement the Syncer feature. However, this project has been put on hold recently because I realized it could provide students with the ability to host malicious content on my infrastructure. Although the code is still up and running, it is not available due to security concerns. I do not want to risk getting banned by my cloud provider because of potentially harmful content hosted by someone else.

Bart: Your student is hosting a dark web site and doing crypto.

Stefan: Initially, I had labs with outbound access to the internet for every single lab. This was because I wanted students to be able to upgrade a Kubernetes cluster, which required access to app repositories or YAML repositories. I had a lab on upgrading a Kubernetes cluster from 1.2.5 to 1.2.6. However, I cut off public egress access due to security concerns. Students could host a miner for a short period, mine some resources, repeat the lab, and install the miner again, posing a significant security risk, this could be mitigated with NetworkPolicy and RBAC.

Bart: Very good. Now I imagine that providing full Kubernetes clusters to students is no small feat. What other approaches did you consider before settling on your final solution?

Stefan: The V-Cluster was a strong one. Firecracker was a thing, but ultimately, full-blown virtualization won. That was the libvirt KVM virtualization, which was provided by KubeVirt. We can talk about KubeVirt for forever, but that technology ultimately won the fight.

Bart: Now, KubeFM is not a technology that all of our listeners may be familiar with. Could you explain how it works in the context of your platform?

Stefan: KubeVirt is another controller in your Kubernetes cluster. When installed, it provides a connection between a Kubernetes pod and a full-blown virtual machine hosted on your hardware. This means you can deploy a custom resource object, such as a virtual machine, which deploys a Kubernetes pod directly coupled with a virtual machine on your host system. As a result, you can use NetworkPolicy policies, expose services via service objects, node ports, and utilize all Kubernetes objects around your virtual machine. In the context of labs, this provides a really isolated environment while still offering the benefits of a Kubernetes cluster. For instance, if a person connects to a lab environment via root, they can access all other IP addresses on the cluster. However, NetworkPolicy policies can be used to prevent this completely, ensuring the virtual machine is completely isolated on the Kubernetes cluster, or rather, the host system, with the frontend being the Kubernetes pod.

Bart: Implementing any technology at scale often comes with its own set of challenges. What were some of the key technical decisions you had to make when implementing KubeVirt?

Stefan: So, definitely, it's a challenging technology because it's complicated. Our KubeVirt is a virtualization, and virtualization is an animal of its own - it's really complicated technology coupled with Kubernetes, which makes it that much more complex.

Bart: Fair enough. Anything that you might want to speak about more specifically about Container Disk and implementing a cache.

Stefan: The main issue with KubeVirt was the technology or mechanism to use persistent volume claims or container disk images. This works by creating an image with the operating system and all the packages installed as a persistent volume claim. If you want to use the same image, you have to clone this persistent volume claim to a different namespace or somewhere else where the student might want to spin up another lab. However, this process took about 10 minutes to migrate a single persistent volume claim, which is not feasible.

I tried to implement a cache of labs initially, but the problem was that the cloning sometimes failed. I had to create a mechanism to remove the persistent volume claim and start over the cloning process, which felt like heavy lifting.

KubeVirt provides container disk images, which work by exporting a virtual machine as a raw file and mounting it in a Docker container under a specific path. The Docker container is then built, and this image can be pushed into Docker Hub, despite being large (8-10 gigabytes). Every time this Docker image is required, it can be pulled from the registry or used from the cache in Kubernetes, and it's up and running in seconds.

Bart: Running full virtual machines for each student environment seems somewhat resource intensive. How do you manage the performance and cost implications of this approach?

Stefan: Well, I kind of doubled into it in the beginning, but I'll tell you how it worked before. It worked by giving you three virtual machines with every single lab environment. I decided to allocate 1.5 gigabytes of memory to the control plane node, which is its own virtual machine, and that seemed to be enough to run all the control plane components. Then, I allocated 1.1 gigabyte for a normal worker node that might come with a lab. Every single lab had three nodes, every single time, because they were in the same namespace and came as a package. So, every time a student clicked "claim lab" or "start lab," they would receive three nodes, which is close to 5 gigabytes of memory, and about three vCPUs. This is a lot of resources. However, I created a mechanism to limit that. The way it's limited now is that labs which require three nodes still have three nodes, but labs that just require a control plane get only a control plane node. Labs that require a control plane and one worker get that. There might be labs that require a single control plane node and then five worker nodes, although I cannot imagine a lab like that, but it's possible in the future. So, the system is really dynamic right now.

Bart: Resource management in a learning platform can be tricky, especially when students might abandon exercises. How do you handle scenarios where students start multiple exercises and then stop using them, when they're not [crypto mining](What is crypto mining in this context? Is it related to cryptocurrency or something else?), of course.

Stefan: A student can only start one lab. If they don't end it, the lab does not time out, and they cannot start another lab. There is a mechanism that always checks whether a student is actively working in the lab or has received credentials, and it doesn't let them spin up another one. This mechanism prevents students from starting multiple labs. Each lab has a time limit - currently, all labs have a one-hour limit, and Labs4Grabs sandboxes have a two-hour limit. If the time is up, a service kills the lab and ends the session. This is how it's handled.

Bart: Now, we've talked about security more than once in this conversation for good reasons, as it is a critical concern in multi-tenant environments. How do you address networking security in your Kubernetes setup? We discussed NetworkPolicy. Could you talk about that? Is there anything else that would be beneficial to know in terms of the setup that goes into this?

Stefan: The initial idea was that the Kubernetes lab consisted of three nodes in the same namespace. It's really easy to set up a NetworkPolicy to cut off a single namespace from the rest of the system. You just create a network policy and allow all inbound and outbound access to the internet, and prevent the rest, apart from DNS, which is necessary for communication with the Kubernetes DNS. In the new system, I have two namespaces: one namespace per control plane and one namespace for worker nodes. When a student initiates the lab, the system claims the nodes first, depending on how many are needed, and then makes the necessary connections between the namespaces and the labels required for the control plane to communicate with the workers and for workers to communicate with the control plane from other namespaces. The Kube DNS is also required for all components to access. The control plane node is the only node exposed to the public. The labs are all cached, waiting for students to claim them or start labs. If a student starts a lab, a node port service is created to allow them access to the system by SSH. I don't pre-create anything, so if you scan the host, only one port is open, and that's the API server. Everything else is prevented and secured until the student starts a lab, in which case the node port is opened, and students can have access to the control plane node. From the control plane node, they can then have access to the worker node.

Bart: Now looking forward towards the future, what developments or improvements are you planning for Labs4Grabs.io?

Stefan: Regarding content, I want to create more materials for the CKA, CKAD, and CKS exams. I have an idea to provide a repository for the community to create their own labs. I have a system with a YAML file that defines checks, node labels, taints, and other necessary components for a lab. I also have scripts that install required components before and after the lab. It would be interesting to see what the community comes up with. Additionally, I think it would be beneficial to provide this technology to companies that want to test their potential candidates. A public repository could make this happen.

Bart: Interesting. To take that further, because you mentioned the process of certifications, there is a debate around this. Is a CKA, CKAD, or CKS certification a good way for a company to know if someone will do a good job? You propose that there could be other ways to do that. What are your thoughts on certifications as a way of validating someone's ability to be a solid engineer when it comes to Kubernetes?

Stefan: A certification is just a piece of paper. It just says that you did this thing, you created some deployments, you created a stateful set, you repaired kubelet, you have a basic understanding of how Kubernetes works. But when it comes to real-world problems, that's lacking. But it's fine because if a company feels comfortable with hiring you based on the certification, such as CKA, CKAD, or CKS, you will gain knowledge, real-world knowledge on the job. You have plenty of resources online when you start.

Bart: Reflecting on your journey developing Labs4Grabs, what would you do differently if you could start over now?

Stefan: Just probably use Kubernetes a lot more, because the way it worked before was by GitHub pipelines, specifically GitHub Actions. When a student performed an action, such as claiming a lab or creating and caching labs, it was all done via GitHub Actions. I would definitely invest more time into creating a Kubernetes service that creates labs, caches them, and has access to Kubernetes environments, rather than relying on GitHub Actions. So, Labs4Grabs no longer uses GitHub Actions; it's now purely on Kubernetes. There are almost 10 services that handle different aspects of labs, but I would start with Kubernetes in mind first, and then it would be easier.

Bart: Now, getting towards the end, we found out that you're into mountaineering and boxing. I don't have climbing gear, but I do have my boxing gloves nearby because I also love boxing.

Stefan: Tell us about your hobbies. Start with mountaineering, and then we can begin a new podcast episode about boxing. Mountaineering in Slovakia is a great experience, particularly in the High Tatras mountain range, which includes the High and Low Tatras. The High Tatras are where the magic happens. I have always been drawn to nature, and I really like it. Unfortunately, I don't have a lot of time for it now due to personal commitments. It's difficult to find time because I have to travel, usually staying overnight, and get up early in the morning to go mountaineering.

Bart: i don't have a lot of time, but hopefully in the future, I can get back to it. If you're ever in the north of Spain, the best country where I live, we have lots of great places for mountaineering. Everyone who loves nature would enjoy it, but I would love to travel to Slovakia to see what you're describing, maybe visit the High Tatras. Now, tell me about boxing. I used to live in Britain and got into boxing, then I

Stefan: It was just something for me to do. I was never a fighter, but I got into boxing for the bags and cardio. I then started sparring and got my face pretty beat up by one guy, so I left boxing for a while. However, after moving to Slovakia, specifically to the area near the High Tatras, I started sparring again and it has become a big part of my life.

Bart: I've been involved in martial arts for quite some time, but for the last few years, I've been focusing on boxing primarily because the workouts are really good. The element of sparring can be violent, but it doesn't have to be; it can actually be something that's really controlled. For me, it's beneficial not just for cardio reasons, as it's very heavy-duty exercise, but also because a lot of it is about emotional control and slowing down my reactions. I find this part to be really helpful for many things, including concentration, focus, and reading situations. In moments where my body is in extreme tension, trying to apply as much calm as possible is a very interesting exercise.

Stefan: The calmer you are, the more you can handle. If you take hits when you're calm, then you're already halfway there.

Bart: It's absolutely true. And I think that sometimes it's a metaphor for a lot of other things in life, where in the business world or the work world, you're going to encounter problems. How you react to them or how you're prepared for them when they arrive is where the difference lies. We will talk a lot more about boxing when the recording finishes.

Stefan: Awesome. Sure.

Bart: That's great. It's really nice to talk to you because people have a certain image of what they imagine combat sports might be like, but the reality is very different. For me, what stands out is the immense amount of respect, trust, care, and support for everyone involved. It's not about trying to hurt someone or make their day bad; it's not about that at all. There are some people who come from different backgrounds and have personal issues, and some tend to have different approaches.

Stefan: To take it up a notch, and I mean, before I was starting, I was kind of scared of those people, and now I'm like, "Hey, let's bring it on."

Bart: It's just another part of the experience. I think the gyms can influence the way it's approached and understood. However, as you said, some people are on a mission, and you must keep that in mind before stepping into the ring.

Stefan: Good way of putting it.

Bart: So, you are quite busy with a lot of different things, but what's next for you?

Stefan: I just want to keep doing what I'm doing. I'm really happy with my current situation. I have a side project, Labs4Grabs, which gives me a lot of skills. I have a great job, working with people who have much more experience than I can dream of, which makes it a great place to be with interesting problems to solve. I would like to keep doing what I'm doing at the moment.

Bart: And what's the best way for people to get in touch with you?

Stefan: Definitely LinkedIn, although I'm not on it currently. I need to install the app on my phone to receive notifications. A great way to connect with me is via email or Slack - just connect to Labs4Grabs Slack, and I'm available 24-7.

Bart: Great. It was very nice having you on our podcast today, hearing about your experience. Keep up the amazing work. We will be looking forward to seeing your next steps.

Stefan: Thank you very much. Thanks for having me.

Bart: Take care. Bye-bye.

Stefan: Bye-bye.