Why Helm's design is flawed

Bart: Helm is a well-known tool for packaging Kubernetes YAML templates and deploying them on a cluster with a given set of template values. In today's episode, we'll be hearing from a person who, despite the many Helm fans out there, is not a fan of Helm and thinks that its design is fundamentally flawed. His name is Jacco. He's based in the Netherlands. He's a CTO and founder and working with Kubernetes for quite some time. And there are some issues that he has with Helm that he shared with us. And we want you to listen to get your opinion as well. This is KubeFM, the podcast that broadcasts the greatest and latest trends in the Kubernetes ecosystem shared by experts who are passionately working on this and sharing their knowledge. Let's take a look. Jacco, if you have a brand new Kubernetes cluster, what are the three tools that you're going to install first and why?

Jacco: Okay. Well, that would be Argo CD, first of all, because I think that's a great tool. And it's a great starting point for if you want to do GitOps, right? So I'm a big fan of doing GitOps and, you know, writing all my clusters in YAML files in a Git repo. And then basically... Argo CD makes sure that it runs on the cluster and that I don't have to do any manual kubectl commands anymore. Well, I hope, you know, hopefully I don't have to do that anymore. But most of that is then automated by Argo CD. Secondly, Prometheus, I think it's really important to have, you know, proper metrics on your server. Look, you want to see what's happening there. that's important and then there's like some kind of ELK stack so you want to know all kinds of logging that's happening on your cluster and i think these are great tools that that help you to get insight in those logings and see what's going wrong these are also pretty popular uh you know pretty popular tools i think they're mature and a good way to start

Bart: All right. Now we got that out of the way. Tell me a little bit more about what you do and who you work for.

Jacco: Okay. Well, I'm a CTO of Vidicrowd. It's a small startup in the Netherlands. We had a bumpy ride the last few years, but we finally get some traction now, which is cool. I'm basically also the developer of most of the platform, so I'm responsible for the programming of the platform. but also for maintaining the cluster for that. And that's basically why I started to learn Kubernetes and Helm. And yeah, that's also why I wrote this piece that we're going to discuss today.

Bart: So tell me about your technological journey and how you ended up getting into Cloud Native.

Jacco: Okay, well, as a startup, you want to use the most modern tools that are out there. And maintaining your own hardware is not 2023 anymore, right? So you want to go into the cloud. And with a small team, you need to be smart. And you need to use the tools that do most of the things automatically without you having to look at it. And that's why I think Kubernetes is a great tool. And, you know, it makes it possible to, you know, to specify your cluster and give it a kick and then it should run, right? And of course you need some, you know, you need to, you know, some tools to... to be able to monitor, but for the most part, it runs automatically. And that's what I like. That's why I invested a lot of time in learning Kubernetes and the ecosystem and the tools around it. And I think it's also very interesting. technology to develop yourself as a developer because development is not only programming anymore just per se, right? It's the whole thing around deploying it, how it should run in your cluster. And I think that's a good skill to have, which is usable in any situation. If you want to move jobs or whatever, it's a good thing to learn.

Bart: Absolutely. In terms of your process learning Kubernetes, what were some of the resources that were most helpful in while you were leveling up? What helped you out the most?

Jacco: I'm just mostly a hands-on guy. So I started with just the official documentation, and just get going and, and, you know, trying and making mistakes and trying again and making more mistakes and trying again, et cetera. I'm not really a fan of doing like courses, like sometimes I do, but I still want to learn it the hard way because I think that's the best way to learn it, you know, to make all the mistakes. And if you do a course, you, you learn the easy path that the hard path is not often not in the course. And, and, and I think that's. those are opportunities for you to learn. And those are often not in the courses.

Bart: And if you had to go back in time to give your previous self advice about how's the best way to learn Kubernetes, would there be anything that you would focus on first or perhaps leave till maybe later on?

Jacco: Well, the way I did is I do this with all the new technologies that I learn. You just stumble upon them. I do some quick assessments. whether it's worth my time to invest, to invest my time and to investigate it, yes or not. And then just start going and find some free time to play with it and learn it. And I always do it that way. And I don't think I would do it differently the next time.

Bart: In terms of what we want to talk about today, you wrote a bit of a controversial article. And could you tell our audience what that article is about?

Jacco: Okay, so the article is about my experience with learning Kubernetes and Helm. I found that Helm was a really popular tool in Kubernetes, but I also found that it had some flaws. You know, in the article, I draw a comparison between Helm and some other popular web technology. It's PHP. It was designed like somewhere in the beginning of the 1990s. And, you know, and I think there's a big parallel between the two because they both take a templating approach to solving a problem. And in general, I don't think using templating is the right way to solve these kinds of problems. And in the article, I argued that also for PHP, there were better alternatives at the time. And still, this templating approach called on. And, you know, there were zillions of packages being built with PHP. It grew really big and kind of the... You know, the momentum to choose for another technology or other approach was like gone, right? It was just too big and nobody would leave it anymore. And it's, you know, and it's still used, right? There are better technologies, but there's still a lot of legacy in PHP. And I think the same thing holds for Helm. It's big. There are zillions of packages. Everybody's using it. But there are fundamental flaws that are still out there. And I think it's not really easy for Helm to solve them. Because if you want to solve these fundamental problems, it's like a completely different thing. It's not Helm anymore. But still, I would love to see another kind of... approach to packaging Kubernetes projects, basically. That's what this article is about.

Bart: It's clear that there are some things you don't really like about it, but tell me what you really don't like about it. What is it that you don't like about Helm's approach? Perhaps that it's not really doing the things that it says to be doing. Tell me more about that.

Jacco: Okay, so what Helm basically does is you write your YAML files as a template. So you insert some commands in your YAML file that do some replacements, right? That's what templating does. But Helm doesn't treat the files as... Kubernetes resources in any way. It even doesn't treat it as a YAML file in any way. It just treats it as a plain text file. And first of all, I think you lose a lot of information there that could be beneficial for the language. One part that you lose is that, for instance, I have some examples in my article where, for instance if you want to insert some values in your YAML file, and this value is a multi-line string basically, then you also have to make sure that you're indenting it in the right way, which I think is a horrible way to write your, you know, write your code or write your template that you have to actually take care of which indentation level you're actually working on and have to code that in your, in your template. I think that's really horrible and really user unfriendly and basically horrible. And it's really, you know, easy to make mistakes there. which makes it, you know, you need more debugging, more testing, et cetera. And you need to test all the cases. I think that's really a horrible experience for the developer. Another example that I show in the article is that, for instance, you have, you know, normal Kubernetes YAML has a very strict schema, right? You have very good specified fields and these fields have very well specified types of values and your editor can help you with that. And if you make a mistake there, you immediately get a proper warning there. So that's really helpful. But as soon as you start using Helm, this is basically lost because your editor doesn't know about the Helm structure and the structure of your values file in your project. I give an example from a project where you want to specify, for instance, the limits on your Kubernetes resource. Now, specifying the limits in a deployment is... is part of the specification of deployment. It's very well-defined and well-structured. Now, if you want to have defined this in your Helm project, then you kind of have to copy this entire schema of specifying limits into your Helm template. And basically, you're reimplementing Kubernetes in your Helm. Kubernetes deployment in your Helm project, which is also a horrible thing, I think, because it's already been invented and you don't want to invent it twice.

Bart: What about Helm packages? Do you think those are bad too?

Jacco: No, of course not. A lot of people spend a lot of time implementing Helm packages and they test them thoroughly and they're being used in production environments for years already, and they're stable. There's nothing wrong with that, right? As long as they're stable, then it's fine. I only think that Helm by itself is not really a user-friendly tool, right? I think it can do a better way in helping you write stable packages, helping you with debugging it. helping you with writing less code and helping you with writing very readable code. And I think in that sense, Helm does a very bad job. But of course, there are many brilliant packages out there and I use them all the time. And I'm not going to reimplement them all just because I don't like Helm, right?

Bart: And could you share any examples of this more specifically?

Jacco: Well, I use, for instance, Ingress NGINX a lot. I think it's well written, right? It's well tested. There's nothing wrong with that. Or the kube-prometheus-stack is also a good example of that. It's a big project. There are many resources there. Documentation is a bit lagging, but in a sense, I think it's a brilliant project. But I would hope that there would be a better tool for the developers of those projects so that they would have an easier life. So I want to give an example, for instance, from NGINX Ingress. And what you see there is that it kind of mimics the standard Kubernetes deployment schema regarding the liveness probes and the readiness probes. These are very well defined schemas in the original Kubernetes deployment resource, but the developer of the Ingress Nginx template wanted to allow the user also define the liveness probe and readiness probe values. So what he did is he kind of copied the original deployment. these values for the liveness probe and readiness probe into the template. So basically it was re-implementing part of the original Kubernetes deployment resource into his Helm definition. First of all, I think that's just a waste of time because there already is a proper resource. And secondly, you lose the strict schema that the original deployment resource has and that the values file for the Helm schema doesn't have this strict schema anymore.

Bart: So can you tell me then about when was the moment where you realized, okay, I can conclusively say that Helm is wrong. Was it love or hate at first sight, if we want to call it that, or was it something that developed over time?

Jacco: I think I've never been really in love with Helm because I always do some kind of due diligence with the new tool or resource. I want to just make sure that it's worth my time. And I did the same with Helm and what I saw on the internet, on various projects, I didn't really like what I saw. I mean, if you look at the files that are produced by various projects, they're kind of unreadable. And to me, that's like an indication that the technology is not really, either not really complete. Or it's just flawed from the beginning. And that's why I took the approach that, okay, if there's a good project out there that's using Helm, and it's well-maintained and it's stable, fine, I will use it, but I will not write my own Helm scripts, write my own Helm. packages because I simply think it's not worth my time.

Bart: In the JavaScript ecosystem, there's a popular meme with two books, a thick book about JavaScript and a slim manual on JavaScript, the good parts, meant to make fun of what language features you should use or not. Do you think most people should restrict their use of Helm and perhaps limit to using it only as a package manager?

Jacco: Well, I would definitely advise that. I do that in my own projects. I use Helm for the package that I want to import, right? So not my own packages, but packages from other people. But for my own projects, I use Kustomize a lot. I'm not saying that Kustomize is the perfect tool. It has its own flaws. But at least it's strict, it's well-defined, it's well-documented, and it serves the purpose of, yeah, configuring and defining my own projects for the various deployments that I have. And I think it matches very well with the GitOps approach. There's good support for Kustomize in ArgoCD. So that together makes that Kustomize for me is a very workable solution. I don't think Kustomize is flawed. I think there are better... I think there will be some evolution at some point where there are better tools that do the job than currently Helm and Kustomize do. But still, for me, Kustomize does the job in my own projects.

Bart: Okay. But would you say that in terms of the love, Helm has a pretty strong community behind it, a fair amount of interest. Do you feel like Kustomize receives the love that it deserves?

Jacco: It's not as big as Helm is right now, but I see continuous improvements and it is used by a lot of people. So I don't think it will die. It will improve. But, you know, Kustomize is not a package manager and it's not designed to be, and it doesn't, you know, I don't see it as a starting point for writing a package manager based on Kustomize. I don't think that's the right tool for that. So Kustomize is great for basically for your own projects and managing your own projects. And that's what it's great for, not as a package manager.

Bart: And if not using Kustomize or Helm, what templating strategy would you recommend? And do you even, just as a side question, do you even use Helm at the moment?

Jacco: I don't use Helm in my own projects but I do import other people's Helm projects. As long as I don't have to look at it then it's fine. But to start with your first question, I don't think you should see Kubernetes packages as templates. You know, just get rid of the word templates, basically, because they're not templates, they're packages. And these packages define some resources and you should be able to configure those resources. But that's a different thing as templating. And so it's basically the word templating that triggers my fury.

Bart: Okay, good. Speaking of emotions, you just mentioned fury. What was the reaction from the community towards the article that you wrote? You put your criticism out there. Did you receive any criticism back?

Jacco: No, I did not. I did not receive any hate mail or people banging on the door or whatever, or I haven't been blocked on any site. So that's all great. I actually got a lot of positive feedback and people saying, hey, good write up. I agree or people playing with their own projects and I didn't get any critical notes. So maybe, you know, people who don't like my work, they didn't respond or they just ran away quickly. I don't know. But I think I touched upon something that a lot of people actually knew. but they didn't kind of write it down or they just, you know, go with the flow and say, okay, you know, this is big. So it's probably a good thing. So I hope I made a few people think again and, you know, see that although it's a big project, right, there are zillions of packages, doesn't mean that it's, you know, that it's a great, you know, that it's not, that doesn't have any flaws basically.

Bart: Got it. And was that essentially your goal in publishing an article that could be seen as spicy or controversial or polarizing to sort of wake people up to the idea that you don't feel like you're the only one with these opinions? Or what were some of the other goals you had when you wrote the article?

Jacco: Well, I didn't write a lot of articles yet, but I kind of have the idea that as soon as something is growing in my mind, and I feel that it is an opinion that is not alive somewhere on the internet, then you do the research and think, hey, are there any other people that think the same way? And I couldn't find any criticism there. And I found it strange because to me, the flaws are very clear. And that actually made me write it. And it also guided as a good starting point to write the first article. It wasn't my first, but anyway, one of my first.

Bart: Do you anticipate you'll be writing other controversial articles like this in the future? Or what else do you have in mind in terms of your next steps?

Jacco: Well, you know, maybe I want to write an article about why US politics is flawed. But I'm not so sure if I want to do that because I think I will receive some hate mail there. And, you know, I'm not a US citizen. So, you know, why do I care? But anyway, yeah, probably I will. You know, like I said, something is in my mind and I think it's opinion. that I think other people should know about, and I can't find anything, any other people that think the same way, then I'll probably write it down. And that's what I want to do more often.

Bart: And once again, as we spoke in the beginning, helping others troubleshoot, get through some of these difficulties is very much a big part of making it easier for Kubernetes to be adopted, helping people level up. So grateful for those contributions. When you're not looking for controversial topics, it looks like you're into both skiing and scuba diving. Two adventurous activities. Tell me more about that. How'd you get started with both of those?

Jacco: Well actually scuba diving was like 15 years ago. It was my professor at university that did it and I thought, okay, that sounds like a fun activity. So I just took some classes and did some diving here in the Netherlands and abroad. And because it's great, you're really on your own, you can be very mindful when you're deep under the water. And I think that's really relaxing. And of course it's very beautiful under there. And then skiing is really my passion. I just love it to be out there in the cold, up in the mountains. It's perfect. I'm a little bit sad that, you know, climate change is making all the ice melt and snow melt at least here in Europe. Some ski locations are already closing down and never, you know, they're not going to open up this year anymore. So that's a sad thing. So hopefully I can do it for the rest of my life, but let's see.

Bart: If someone has to go scuba diving in the Netherlands, because it's not often that we hear about that as a hotspot, what are the best places to go?

Jacco: Well, there's actually Zeeland. It's not New Zealand, but we have a Zeeland in the Netherlands as well.

Bart: The original Zeeland, the real Zeeland.

Jacco: The real Zeeland, yeah. It gave the name to New Zealand. And it's actually, there's a lot of islands and lakes around there. And, you know, a lot of people in the Netherlands also don't know, but it's actually really nice to see their reefs. There's a lot of life out there. And it's a good place to start diving because it's not really deep. Sometimes the water can be a bit rough and, you know, what's that called? Not really transparent.

Bart: Yeah, like murky.

Jacco: Murky, right. Yeah. But it's a great place.

Bart: Did not know that. I've been in the Netherlands several times and I was yet to find out about scuba diving there. So that's good to know. Last but certainly not least, for people that want to get in touch with you, whether it's about scuba diving, skiing or car traversal takes around Helm, what's the best way to do so?

Jacco: Yeah. So, well, just find the Medium article and drop a note there. You can find me on LinkedIn just under my name.

Bart: Perfect. Well, thank you very much for sharing your time and your thoughts today and look forward to crossing paths with you soon.

Jacco: Okay. Thank you. Take care. Bye-bye.