Simplifying Kubernetes deployments with a unified Helm chart

Bart: Managing microservices and Kubernetes at scale comes with a unique set of challenges, from inconsistent Helm chart configurations to the complexity of maintaining best practices across multiple services. In this episode of KubeFM, we're joined by Calin Florescu, a DevOps engineer, to discuss a unified Helm chart approach that simplifies microservices management. We'll explore how standardization improves deployments, the trade-offs between flexibility and control, and the role of automation in documentation and testing. If you're working with Kubernetes and Helm, this conversation offers practical insights into streamlining operations and reducing overhead. This episode of KubeFM is sponsored by Learnk8s, which has been providing Kubernetes training to engineers all over the world for almost a decade. Courses are taught in-person or online to groups and individuals, and are instructor-led, with 60% practical and 40% theoretical content. Students have access to materials for the rest of their lives. For more information, check out learnk8s.io. Now, let's get into the episode. Welcome to KubeFM. Thanks for joining us today. Before we get into the core of the episode, I want to address this question: What are three emerging Kubernetes tools that you're keeping an eye on?

Calin: The main things I'm looking for now are Crossplane, Karpenter, and KEDA. I see Kubernetes as a platform, not just a container orchestration tool, because it has evolved with the support of the community and the development of various tools. This evolution allows users to do almost anything they want. Crossplane empowers engineers to manage their own infrastructure using a common language. Karpenter is a tool that solves the problem of node sizing, eliminating the need for "t-shirt sizing" for nodes, which is particularly useful in microservice architectures with services of different sizes. KEDA is used for auto-scaling, and I'm interested in exploring the possibilities of auto-scaling based on metrics, as CPU or memory metrics are not always sufficient for complex applications and businesses. The Horizontal Pod Autoscaler (HPA) provides a way to scale based on metrics, which is a useful feature to have in mind. These three tools are the main areas I'm focusing on.

Bart: And what do you do and who do you work for?

Calin: I describe myself as a platform engineer, rather than just a DevOps guy, because it best suits what I do. I help companies and clients achieve DevOps practices and introduce them into their development process, allowing them to increase speed, reliability, and reduce costs. My focus is on helping people improve and be better than they were yesterday. Currently, I am an infrastructure consultant for Thoughtworks. I got into cloud native through a unique experience. Before working with cloud native and DevOps platform engineering, I was a programmer and engineer, working extensively with JavaScript as a full-stack engineer, using Node, React, and other popular technologies at the time. I was hired for a small project with a tight deadline and limited personnel, and no one was willing to work on Docker and Kubernetes. As a result, I had to take on the task, and it was a forced transition. I dove in headfirst, and that's how I ended up where I am today.

Bart: And in that process, and now as well, how do you stay up to date with all the things going on in the Kubernetes and Cloud Native ecosystem? What resources will help you as you head into it, and what are some of the things that you still use nowadays to stay up to date, such as the Kubernetes Blog, or other resources like Learnk8s?

Calin: So, to me, the main Kubernetes Blog is a thing that I'm subscribed to, and I'm looking for news with every minor and major version that happens in the environment. I like to stay updated on what's happening there. I also approach things by learning from the people around me and their experiences. I have many DevOps engineers and Kubernetes engineers connected on LinkedIn, which is the main platform I use. This is how I found the Learnk8s community and page, which I subscribe to and enjoy following. My approach is to tackle a problem or task at work and then start researching the web for solutions, tools, or plugins that can solve the issue. I take a hands-on approach, trying to minimize reading things out of curiosity, because I believe that if you don't practice something, it's hard to learn or internalize. That's my approach.

Bart: If you could go back in time and share one career tip with your younger self, what would it be?

Calin: Definitely stop context switching that much and set up time blocks for deep focus. In this field, it's easy to get a lot of messages from people saying something is not working or there's an issue. Because I like helping people, I try to make their lives easier first, and then mine. This is probably why my work progress or efficiency is not always 100%, maybe around 90 or 95%. Time blocking and practicing deep focus periods, such as the Pomodoro Technique or Deep Work, is advice I would give myself.

Bart: To take that point a little bit further, since you mentioned focus and deep focus, have you read anything by Cal Newport, such as his book Deep Work?

Calin: Funnily enough, that's the book I read, 'Deep Work' by Cal Newport. I read it about two years ago, and it changed the way I work. When faced with bigger challenges, I use time boxing and restrict myself from any surrounding sounds or noise, like messages. This helps me focus a lot, similar to the Pomodoro Technique.

Bart: Calin Florescu is discussing Cal Newport, a programmer and lecturer in the United States, who has written about the topic of living in a world full of distractions, such as notifications, messages, email, and phone calls. Cal Newport explains that people suffer from solitude deprivation, not having enough time to be alone with their thoughts. It is assumed that this is the way things will always be. However, the host wants to know if there are any practical tips for putting this theory into practice, especially for someone who suffers from context switching and has been diagnosed with ADHD. The host is looking for tools, techniques, and tips to help others who want to approach things in a similar way, beyond just the theory of timeboxing, which is related to the Pomodoro Technique. One of Cal Newport's notable books on the topic is Deep Work.

Calin: I'm very self-aware about myself, which is a skill of mine. I notice when something goes wrong or when I want to do something, and I actually know all the steps to do it. To improve focus, I would recommend working using the Pomodoro Technique, which involves 25 minutes of work with breaks. This is a good start. Cal Newport promotes two to four hours of deep focus per day, as discussed in his book Deep Work. Another strategy that has worked for me is rescheduling my work to make better use of my time. I've become a morning person, waking up early when there are fewer distractions, making it easier to focus. Initially, I dedicated my mornings to learning and practicing, and over time, I was able to schedule time boxes throughout the day. I would advise everyone to try the Pomodoro Technique first, as it's easy and provides a timed alert, and then consider adjusting their schedule to optimize their focus.

Bart: Great. Now, as part of our monthly content discovery, we found articles about Streamlining Microservices Management: A Unified Helm Chart Approach and a unified Helm chart approach. We want to dig into this topic with you. Let's start by discussing the challenges of managing microservices in Kubernetes at scale. What specific issues have you encountered when dealing with many services in this environment?

Calin: To start explaining the root cause I found, I think we should discuss the team's architecture that is common across all companies. From my experience, companies either have a central platform team that takes care of all the infrastructure, with multiple engineering teams using that infrastructure and connecting to it, or they have a dedicated DevOps team that takes care of the ownership of the resources that team is using.

The problem appears with the central platform team, where the team controlling the infrastructure creates templates, but as the business scales, the teams need to manage more infrastructure. The engineering teams should focus on creating the best product, writing efficient code, and making the best decisions for the product. Their focus is not on running services in Kubernetes or having the best templates.

As a result, the problem arises when multiple teams work together, and all their microservices end up in Kubernetes. While they all use deployments, secrets, and config maps, they are often written differently. This is because the engineer responsible for implementing these services may not have the knowledge to apply best practices, and it is not their primary responsibility. Consequently, when something breaks, someone from the central platform team needs to intervene and figure out what happened.

This leads to a situation where the central platform team needs to be familiar with multiple different approaches, almost like speaking 20 languages at once. Additionally, context switching is required when working from service to service, as each may have been deployed, released, or managed differently. I believe this is one of the problems I discussed in my article, Streamlining Microservices Management: A Unified Helm Chart Approach.

Bart: You mentioned inconsistencies and how services are written. How does this focus on code efficiency over Helm template optimization impact the overall management of large-scale Kubernetes deployments?

Calin: The main issue with this is time. All of these inefficiencies are consuming valuable time. The idea is simple: you want to get your product to the market as fast as possible, get feedback from your customers as fast as possible, implement that feedback, and the cycle continues. These inefficiencies consume time and make everything slower. If it takes a long time to create a new release and make it available in production, you are losing money. This is how it translates into practice and what's actually happening. You can look at the code and think it's not that important, but if you take a deeper look and try to see how this is impacting your business, you can see that it delays you a lot.

Bart: To address these challenges, you proposed a unified Helm Chart approach. Can you walk us through the solution and explain how it works in practice?

Calin: I noticed that in a company with multiple teams, all the services are using the same resources, Kubernetes objects, or APIs. They are all using different deployments or StatefulSets, config maps, or secrets, with minor differences between them. I thought that templating could fix this issue, as Helm is good with templating. By doing the templating in a central place, with all function definitions in one location, engineers could focus on adding input variables to those templates to get the resources they require. In practice, this is quite straightforward. It involves creating a central chart with all the templates, and engineers can import this chart into their repositories. This way, they do not have to write the actual YAML templates for deployments. Instead, they focus on introducing the correct values based on the documentation, as they know the values required by the service, such as CPU requirements or the need for a persistent volume. I aimed to minimize the knowledge level required for engineers to operate in Kubernetes, and I believe this solution achieves that.

Bart: Your solution includes a feature for overriding functionality, which might be a more advanced topic. Why is this important and how have you implemented it?

Calin: I think it is important because there's a fine line between restricting people and empowering them. When you talk about a central approach, it's basically the view of one person, which is me in this case, because I built it. I don't want that because, at some point, it can become restrictive. In some teams, there are people who know Helm, YAML, and how to work with them, and they have the power to do so. I try to be as flexible as possible while also implying best practices and imposing some guidelines. That's why the override functionality was introduced. I didn't want to be a bottleneck for everything, so if someone needed something from the chart to be added, they wouldn't have to come to me. If they have the power and knowledge, they can do it themselves.

This Helm chart operates on functions defined for each Kubernetes object, which can be seen in the GitHub repository. Each object has a corresponding override function. This creates two dictionaries that are merged with another function. The main template has de facto guidelines, and on top of that, you can define another dictionary. At the end, the two dictionaries are merged, creating the end resource that can have input from the central chart and also input from the engineer operating it.

Bart: Documentation is often a significant challenge in software development. How have you addressed this aspect in your unified Helm chart approach?

Calin: So, I tried to implement something that works automatically with minimal input from my side. I found a tool online called Helm Docs, created by Norwood DJ, on GitHub. This tool works with the chart by introducing a pre-commit hook that runs every time I make a change to my repository. Based on comments I add to the value files of the chart, it prints out markdown documentation that everyone can read. I focused on this because change can be difficult for people to implement, and as an engineer, you don't want to spend a lot of time refactoring something. That's why this documentation is important - I want it to be easy and straightforward. I wanted something automatic because the value files or definitions for this chart can get really big, with each object having its own inputs. Writing it manually in a markdown file would take a lot of time, probably more time than implementing the whole idea. This is how I sorted it out.

Bart: Testing is another crucial aspect of software development. How have you incorporated testing into this unified approach?

Calin: Testing was done with a plugin, a Helm plugin, the Helm unittest. This is the automatic part where I implemented some unittests in the chart. After creating a new template for a different resource, I added a unit test to check the template and ensure everything works fine, that everything is rendered correctly, and the resource has the required fields based on the Kubernetes API. This is how I did it during the development of the chart. Another way to test this is with real-time practice, because you cannot just write templates and tests and then leave it to the developers to implement without potentially breaking something, especially in a live environment where everything is working. So, each time I added or refactored something, I conducted tests myself on a dummy repository where I created some resources. That's how I tested it against the Kubernetes API.

Bart: While this approach seems beneficial, it could potentially limit flexibility for individual teams. How do you address concerns about over-standardization and loss of team autonomy.

Calin: Overriding is one of the key aspects. Another important part, which can be seen in articles or GitHub repositories, is the social skill of communication. From the beginning, I invited everyone to collaborate, join, and share their needs. This is why the version on GitHub is different from the one I created for my client, as the latter is specific to their business model. It includes numerous automations built based on feedback from engineers. Having a feedback loop is essential; it involves building, getting feedback, refactoring, and rebuilding. Constant communication with stakeholders, in this case, the engineers, allows the solution to remain flexible and not overly restricted.

Bart: A centralized Helm chart could potentially create a single point of failure. To mitigate this risk and ensure changes do not negatively impact multiple services simultaneously, consider implementing version control and change management, adopting a modular chart design, utilizing automated testing, deploying canary releases, setting up monitoring and alerting, and maintaining thorough documentation and communication.

Calin: That's a really good question. I manage this through versioning, having a versioning system with major releases, minor releases, and patches. This helps a lot, especially since you can specify the version to use for a dependency in a chart or how to get the latest version. The options allow you to take the latest patch, the latest minor, or the latest major whenever you specify a version. Through versioning, I managed to fix this situation. Another way to address this is through testing, conducting thorough tests on the chart and smaller services I'm working on. After completing the testing, a new version gets released, and everyone can collect and benefit from it.

Bart: Versioning and access control can become complex with a unified Helm chart. To manage these aspects, particularly when different services require different versions of the chart, consider implementing a robust version control system, using modular charts, managing dependencies, implementing access control, leveraging automation and CI/CD, and maintaining thorough documentation and communication.

Calin: Having a strict version enforced whenever a dependency is defined is one way to take care of the versioning aspect, which differs from chart to chart. Regarding access control, it is platform or vendor agnostic, allowing storage on GitHub, GitLab, or any other private registry. Access was managed through job tokens in GitLab, enabling every created pipeline to access the repository and its container registry where the chart was located, thus granting permission to fetch it. Engineers also had access on their local machines to fetch the repository and install it, using personal access tokens, potentially utilizing a package manager like Helm.

Bart: To wrap up could be removed for a more direct question: Could you summarize the main benefits of this unified Helm chart approach and its potential impact on DevOps or platform teams working with microservices?

Calin: The main impact on engineers is that they can focus on the code they are building or the product they are developing. They do not need to learn new technology; instead, they can focus on an existing template and learn how to work with it based on the documentation. The second benefit is the implementation of best practices and security aspects in this unified Helm chart. Changes are made in one place and distributed through all services using the dependency model. This allows a single DevOps person to serve multiple teams, eliminating the need for multiple DevOps personnel. The main benefits include gaining time and the ability to focus on other important things instead of debugging a small template on an unfamiliar service. The centralized approach ensures consistency across all services.

Bart: Since we're talking about Helm, in one of the episodes from our first season, we had a guest, Jacco Taal, who wrote an article titled "Why Helm's Design is Flawed". He critiques Helm's templating approach for Kubernetes deployments, arguing that Helm's use of templates leads to complex and unreadable YAML configurations, making them more likely to have errors and be difficult to maintain. He also said that Helm's template language lacks awareness of Kubernetes resources in YAML structure, resulting in suboptimal design decisions. He suggests that better alternatives were available at Helm's inception and emphasizes the importance of thoroughly evaluating new technologies before adoption, especially those relying heavily on templating. What are your thoughts on this? Do you agree or disagree?

Calin: I totally agree with what he said, that you can get to a point where everything is so complex and poorly maintained, making it really hard to make any changes. I think this happens with every templating tool or language - you get to a point where it solves the problem, but these are just tools. It's like a hammer or a screwdriver; the people using them are the factor that influences whether something goes well or badly. You can't say that the hammer is bad; it's how you use it to hit the nails that can be bad. If you have people who know best practices about writing code, writing templates, and documenting things, I think you're in a good position. This idea is related to the concept of a unified time chart, which I first heard about three or four years ago from a colleague. I'm not taking credit for this idea; I was just trying to help people who may be in the same situation with my article, Streamlining Microservices Management: A Unified Helm Chart Approach. Having a centralized place or centralized system helps to ensure that the people working with the template are actually qualified to do so and have the knowledge to work with them. This way, you can keep things under control and still have good templates, even if they are complex. As long as everything is documented, it shouldn't be a problem. There are alternative technologies, and they will evolve. I don't doubt that, so maybe Helm is a thing of the past, but I'm eager to adapt when I see something better. The main point I want to make is that there's a difference between tools and the people who use them, and usually, the people are the problem, not the actual tools. You recently participated in a workation.

Bart: The Base Seven community. Can you tell us more about this concept and how it differs from a regular work week or a vacation?

Calin: Workation is a really nice concept that I love. I've been part of it for the past two years with Base Seven, a freelancer community in my city. They do a lot of things. The main benefit is that you work remotely, which is different from working from home. You work from a spectacular place in nature. The last one was in the Danube Delta, a beautiful place full of nature. Changing your context and taking a break from your usual setup helps you see things differently. This is a plus. You also have various activities like going by boat, biking, or hiking, which helps you break your usual routine. The work aspect involves bringing together many freelancers who may not know each other and having them work together in the same place. Inevitably, after a week, everyone talks to each other, and you get new ideas. Even a front-end developer can have an idea that solves a debugging problem you've had for months just by talking to someone. This is because freelancing and remote working can lack social interactions and team collaboration, which workations can fix.

Bart: Great concept. I really like that you mentioned front-end developers are also people with good ideas and contributions. This is a shout out to all front-end developers. The concept sounds really cool and changes how you see things. As a freelancer of almost 10 years, I have noticed a difference from one place to another.

Calin: I'm a firm believer that if you can get rid of the small things that make a difference in our work, we are all doing the same thinking process. No matter if you are writing CSS, a backend engineer, or a database engineer, if you remove the technology and context in which you are working, we all make the same decisions and think in the same way. This is a powerful concept that helps you see everyone as equal, no matter the context, because we all think similarly and may have the same breakdowns.

Bart: There's a lot to be said for that. Now, what's next for you?

Calin: I'm looking for new challenges in the future, trying to see what problems will come my way. I think of myself as a problem solver, which is what I enjoy. I'm doing cloud native because it presents a lot of complex problems. I'm looking for new challenges and helping people, including doing mentorship for those who want to be DevOps or platform engineers. I want to continue doing this because I believe it's essential to give back, considering I didn't get to this point alone and had help from others.

Bart: And what's the best way for people to get in touch with you?

Calin: I'm really active on LinkedIn. You can find me there. It's Calin Florescu. Also, you can reach me by email. If you want to discuss something, feel free to contact me.

Bart: Fantastic. Thank you for joining us today on the podcast. Keep up the amazing work and I hope our paths cross soon in the future. Take care.

Calin: Thank you, Bart, for having me. See ya.