The good, the bad and the ugly of templating YAML in Kubernetes

Bart: What's your experience with YAML and templating Kubernetes? Alexander Block, also known as codablock, kindly shared his knowledge about Helm and its limitations. We've heard about Helm's design being fundamentally flawed before, but he takes it a little bit further, talking about the deployment tool he's created called kluctl, which helps with deploying Kubernetes resources and provides a simpler and more flexible approach to templating to avoid overusing. This episode of KubeFM is brought to you by Komodor. Komodor is a Kubernetes management platform that enables large engineering organizations to maximize the full value of Kubernetes. And how do they do it? By simplifying cluster operations, policy management, troubleshooting, and cost optimization. Komodor makes maintaining continuous Kubernetes reliability at scale possible and fosters a healthy culture of ownership and collaboration. Alex, for our warm-up question, what three emerging Kubernetes tools are you keeping an eye on?

Alex: I don't have a full list, but one thing is Cluster API, which I keep watching what's happening there. Then, not a specific tool, but I'm keeping an eye on the topic of... does everything really have to be CRD and an operator? It looks like it moves in this direction, and I'm interested if it will keep this way. And I watched the development in regard to non-cloud Kubernetes, so not AWS, Azure, or GCP based, but where you really just get a Kubernetes cluster. And don't have to worry about the cloud details. Not sure if it's clear what I mean by that.

Bart: Yeah. No, that's great. Good. And once again, it's just based on your experience, the things that you're noticing, the things that are catching your attention. So for a quick introduction for our audience, can you tell us more about who you are and what you do?

Alex: My name is Alexander Block, codename or nickname codablock on the internet. I'm a freelancer consultant specialized in software development and DevOps, cloud engineering, and everything around that. And I'm the maintainer of kluctl, kluctl, whatever you want to call it. That's basically me.

Bart: And so for folks that don't know, what is kluctl, kluctl, or just plain old klu? What is it and how does it work?

Alex: It's a deployment tool, so it's really focusing on the act of deploying stuff. It does this by gluing together existing stuff from the Helm ecosystem, from Kustomize, and putting templating on top of everything and the project structure to properly structure your project. With a CLI to actually deploy stuff and do diff and dry run and prune, everything that you need that's somehow related to deploying it. It's not a Helm replacement. That might be the first impression if you read about it. It's definitely not a replacement for Helm because it uses Helm extensively in the form of consuming existing Helm charts.

Bart: Fantastic. And in terms of your background, how did you get into Cloud Native?

Alex: By accident. So this was definitely not planned. Maybe I can describe what I did before Cloud Native. I'm a self-taught software developer engineer. I started as a professional C++ games developer. That was my first thing I did when I was 17. As a freelancer, actually, already in the beginning. I moved to mobile games development and specialized in engine development and the platform required to build the actual games. So I was not building the games, but what is required to build the games. I then moved to backend development with Java Enterprise, some C++, and so on. I got very frustrated when I realized that releasing stuff is too slow, like a lot too slow. Too much manual stuff is required. Accidentally got into the DevOps sphere. So Docker came around at that time, and it was clear for me, this is what I need to improve, because this is what frustrated me the most. In the end, I realized cloud and Kubernetes is the only way to fulfill my own requirements and standards. So that's where I am right now. Now you're a freelancer but on top of that, it's a very active ecosystem and so in order to stay on top of things, how do you stay up to date with all the things that are going on in the cloud native and Kubernetes world? Mostly by just watching, consuming Reddit, Twitter, Googling a lot, watching Discord and Slack. I'm not engaging so much in discussion, but I'm watching a lot of stuff. And based on that, I kind of get a feeling what is interesting, what is evolving to be the next thing. And then I just Google for it and read more and more. So I don't have a specific source to get up to date. But like you said, just being active, observing, taking these things into consideration, keeping your eyes open, and if you could go back in time and give yourself one piece of career advice in the past, what would it be? Keep going because I'm actually quite happy with what I did so far. I mean, one or two things could have been improved, but it's part of what happened. So just keep going.

Bart: Also, if we're talking about YAML configuration files, we're going to be talking about kubeadm that's going to come in there too. Can you let us know about your experience with YAML and Kubernetes, Ben?

Alex: So first, fully agree, you cannot escape YAML. And it's basically the language we agreed on in Kubernetes. Compared to other solutions available at that time, I believe it was the best choice. Because what else? XML? No, JSON has its own issues. So I still like it a lot. In my opinion, it's not as bloated as other solutions. For example, XML is seriously bloated. It's a lot easier to write YAML. It's a lot easier to read as a human. So it's not so much meant to be read by a machine, but also meant to be read by a human. I like YAML because you can learn it without becoming a programmer. So you don't have to learn scripting, you don't have to learn... I mean, templating is something I will, of course, also cover later, but you don't have to learn it. You can just learn YAML by repeating what you already see somewhere else, for example. I think you can compare it to... people sometimes say I am an HTML programmer. But HTML is not really programming, which is a good thing because you can learn it without being a programmer. And I think YAML is kind of the same. You can learn it without being too technical. In my opinion, when it comes to the bad things in YAML, it's not YAML itself, it's the tooling around it that might make it worse than it should be. Templating is a good example. It can turn it into a really great thing or a total nightmare. So yeah, that's my point on YAML.

Bart: Could you dive into that a little bit more? When you say templating, what do you mean by that?

Alex: Templating doesn't necessarily mean that you have a real templating language, at least in my opinion. Templating can be replacing everything that starts with a dollar sign and some identifier with some other text. In my opinion, that's some form of templating, even if you wouldn't call it this way, Poor man's templating, whatever. It can be a real templating language, like known from Helm, Zico templating, or Jinja templating, which is what I actually prefer. So, templating is basically replacing some text with other text. And best case, even having some control structures and loops and so on.

Bart: All that sounds really good. I mean, so far, in terms of everything you're saying, that sounds really good. But is there anything, are there downsides or disadvantages of templating?

Alex: Of course. Like everything, it can be overused, extremely. So if you end up having some YAML or whatever else, it doesn't matter. And the amount of templating, the relation between, or the ratio between templating and the actual stuff that you're doing, can get kind of out of control. It can, so, I mean, it's obvious it can get horrible if every second line is templating instead of what you're actually trying to do. That's one thing. Templating can also be overused because, for example, tooling doesn't allow you to solve things in different ways. It might force you to use templating when another solution or another feature in the tooling might have avoided the templating completely. Then, in my opinion, it really depends on which templating language, for example, is used. There are some which are better, at least in my opinion, more understandable and readable by humans, and some which are not. As an example, Go, in my opinion, is really bad for humans to read or to write it, because it does things in ways that one would not expect. As an example, logical operators in the Go templating language are written in a way that no other templating language has ever done. And no human would ever write down on a piece of paper, for example. You wouldn't write and, a, b. You would write a and b, for example, to express logical stuff. So it feels that the Go Templating language is written to be very efficient in reading and rendering it, which I as a human don't care about because I want to write it actually. Another thing specific to Go Templating and Helm is that documentation and perceived best practices can easily lead to overcomplicating, white space handling, or other best practices, which are technically not really required, but everyone ends up doing it for reasons that I personally don't understand, because I don't care if the rendered YAML, after the templating is done, produces two empty lines. I really don't care, because in the end, I want just valid YAML that I can send over to Kubernetes. That's all that I care. It feels like, because Helm was the first one to actually use templating with YAML, and it was the first one having success with it, having brought that option. For me, it feels like templating in YAML somehow now leads to so much negative emotion because it's really not fun to write Helm templates. And people think that this is because templating YAML is bad. But in my opinion, it's not that templating YAML is bad. It's the combination of all this leading to the bad templating.

Bart: It's interesting that you mentioned that about Helm because in season one of KubeFM, we had Jacco who raised similar points about Helm not necessarily being the right tool to template YAML files. He compared Helm templating to PHP and brought back memories of it once upon a time a long time ago. Do you feel like we are stuck in the PHP era of templating?

Alex: Luckily, I missed the PHP era, so I cannot really compare. I also just heard stories of people being stuck in that era. But maybe. I don't know. As I said, it's hard for me to compare. In my opinion, when it comes to the amount of templating in YAML files or the ratio between them, it really depends or it should depend. That's what I mean. It should depend on the type of chart that someone writes. I think there are, I would say, three types of charts that can be written. The first one is public and heavily used charts, so stuff that is meant to be released and consumed by other users. Bitnami charts is a very good example. There are a lot of them. And what they try is they try to kind of serve all use cases and all corner cases of all users. That means if they want to have success, they have to overuse templating because that's actually best practice in that case. So if you look into the Bitnami chart, and observe that every second line is templating, that's actually good because that is best practice in that case. But then there is another type, or the second type of chart, which is kind of the internal and private chart that someone writes just for internal use inside a company, for example, or for some hobby projects. In that case, there is no need to serve all use cases and all corner cases and just everything, which means you actually don't have to use so much templating. So in that case, it shouldn't be that bad. But it feels like it's communicated that best practice is to do it the same way as these big successful charts do it. And people tend to overuse templating in these cases as well, even so it's not required. At least that's my impression. I might be wrong here. Then there's the third type of chart, the glue or typically called the umbrella chart, which doesn't have the purpose of packaging a single application and releasing it in some way internally or publicly, doesn't matter. It's just there to glue together existing charts through dependencies. So let's say you have your own application and it needs a database, it needs Redis, it needs an ingress controller, something like that, and you use an umbrella chart to put them all together and deploy them all at once. So that's the third type of chart that I'm referring to. In this case, you need templating just for the glue. To be honest, I'm not sure how good that works out because I stopped doing that pretty much in the beginning already when I tried it out.

Bart: So what's the alternative to including all possible variations for all variables that you want to customize in a Helm chart?

Alex: I think it's actually impossible to have all variations and all variables that someone might come up with when consuming a Helm chart. So, in my opinion, it would be better to not try that, to not even begin doing that, but instead offer a capability while installing a Helm chart, for example, to patch existing Helm charts. For example, tolerations, label selectors, resources, security-related stuff, it's all very specific to the consumer of the Helm chart. My requirements might be completely different than your requirements, for example, even if we use the same Helm chart. In my opinion, very often it shouldn't be the author of the Helm chart making everything possible, but instead, the tool should be, or Helm in this case, should be able to provide a simple way to patch this in. As an example, it's a feature that Helm could have to give it a list of patches alongside the values, for example, through a patches.yaml file or something like that. I think that if something like that existed in Helm, a lot of the overuse of templating could have just been avoided because there is no need to have so much templating anymore. I'm aware that there is a post-rendering feature in Helm. So you could, for example, use Kustomize as a post-renderer to introduce patches. The problem is this helps you on the consumer side of the Helm chart, but it doesn't alleviate the author of the Helm chart to actually still implement these features, because the author cannot make any assumptions on how you are going to use the Helm chart, which means he has to revert to using templating to make everything possible.

Bart: So, is it safe to say that Helm is safe when it comes to structuring charts?

Alex: It's interesting because in Helm Charts, you have the templates folder, which allows subfolders. So in theory, you can structure your Helm Chart in some way, for example, having subcomponents and so on. Now, the problem is that even though you have these subfolders and they are supported, Helm still treats them as a completely flat structure. So it's the same as if they were in the same folder. That means the way you structure your project has no effect on templating. It doesn't alleviate you from overusing templating in many cases. As an example, if your values can enable or disable a feature or components, you cannot say based on that value, I exclude a folder. That's something you cannot do because you still have to do the exclusion on the manifest level. So you can't do manifest. We'll have to wrap everything around a conditional, and that actually checks for that feature, which doesn't sound that bad in the beginning, but then you look into real-life examples, for example, the Helm chart, we see like 12 subcomponents and every one having a lot of manifests and every single manifest having a set of conditions on top every condition being different and different combinations and corner cases and so on. For example, is a subcomponent enabled and is a secret provided? In that case, the secret has to be created and so on. So things get really complicated after some time. In my opinion, that's not a problem of templating. So templating does not introduce that problem. And templating is not the fault for overusing templating in that case. Because the tooling does not allow to avoid templating completely, it could do so by allowing to exclude folders depending on condition, just as an example.

Bart: So in terms of getting to the next point, if there are questions as to whether or not Helm is safe, like I said, it's safe, a good choice. If we want to take that further, what about values.yaml? Can we put them in the same category regarding being safe, trustworthy, a good option?

Alex: One issue, for example, that I see with values.yaml is that you cannot use templating in it, which I assume is a technical limitation because you are loading the file that defines how you template it. Of course, it's difficult, if not impossible, to implement. But it could be implemented if, for example, you would have something like loading multiple values.yaml and every additional one being loaded, being allowed to use variables from the previous ones and those allowing templating in the various YAML. So it could have been implemented, but it's not. And what I see is it leads to a lot of templating that then has to be moved into the manifest. A lot of complicated conditions and stuff being repeated and just unnecessary complexity that I believe at least partially comes from the values YAML not being able to be templated. There is a TPL function, so you can do some templating, or at least, I'll explain it. You're not really doing templating in the values YAML, but you are defining a string which contains a template that can later be templated in the manifest. Some things can be done, but not everything. It's still quite limited.

Bart: Okay. It looks like you've spent a fair amount of time working with Helm, exploring some of the rough edges, more than the average engineer. When was the point you said, okay, I think I've had enough? Is there anything else that I could possibly use?

Alex: To be honest, I didn't use Helm that much because I decided early on that I'm not going to go through the pain of writing Helm charts for whatever reason. So it's one thing to consume Helm charts. I really love it. You have a lot of Helm charts available. You have a lot of Helm charts from Bitnami, from other vendors. Upstream Helm charts from the creators of the applications that you want to deploy are great. And I really like using them. But when it comes to writing them, in my opinion, it's hell. And I didn't want to go through that pretty much from the beginning. I did it for some time to write the umbrella chart, to kind of orchestrate and glue together all the Helm charts that I wanted to consume. But I realized pretty early that this has so many limits and so much pain that this is not the way I'm going to do it. That I would continue. At least at that time, for me, there was no proper tool to orchestrate and glue together the installation of Helm charts. This is what I was mainly interested in. I was not interested in writing my own Helm charts, but in consuming them in a proper way. I was not aware of helmfile at that point in time, to be honest. If I would have known about that tool, I might have ended up using it and not working on klu.js. I don't know. That might be what had happened. But instead, I started my own tool. But not immediately. It was like... I started with a bash script in the beginning. Maybe everyone has done that, writing a bash script and believing that with bash you can kind of orchestrate your Helm chart installations in a way that is good enough. Of course, I realized, no, I cannot. It will get so complicated that I should not continue doing that. So I started using Python for the same thing. After some time, it turned out that what I wrote there is actually almost capable of deploying everything that I want, so all types of applications. So I rewrote the script in a way that just made it actually possible. After some time, I made it open source. This was version one and everything after that release. After some time, I realized that Python is not the way to continue, especially when it comes to all the integrations of all the tooling in the Go ecosystem and the Kubernetes ecosystem. So I decided to rewrite the tool with Go and release version 2. That's also when I started to promote the tool in different channels. Maybe it's also interesting. If you know that story, how it came to the tool, that also explains why I'm using Jinja2 instead of Go templating. I started with a Python script that made it obvious that I'm going to use Jinja2 for templating because that's what you do in Python. Then when I migrated or when I rewrote everything in Go, I realized I have so many deployments already written. I had to make a choice. Should I rewrite all my deployments to be in Go Templates or should I make it possible to use Jinja 2 in Go. And my decision was to use Jinja 2 in Go to make that possible because I also didn't like Go templates and I was really not happy about using that instead of Jinja 2. So then it came to what it is today. klu CDL after version 2 being in Go with Jinja 2. Today it supports GitOps, so you can also use the same deployments that you have locally in a GitOps fashion, so it can be deployed in a whole workflow while still being able to revert back to push-based workflows whenever you need it. For example, to do dry runs, which is something a lot easier doing locally, or actually a lot more useful doing locally because you can change two lines of deployment manifest, do a dry run, and you know immediately if everything is fine or not, instead of pushing it.

Bart: Very good. Now, I think you mentioned it earlier in the podcast, but I just want to circle back on this. When you were in this process, did you take a look at tools like Kustomize? Did you give them a chance to try it out to see what they had to offer or maybe some of the things that they weren't covering?

Alex: At the same time, when I started working on the first versions of the script, I used Kustomize to have the basic project structure with bases and overlays and everything. In the beginning, I had a lot of hope when I read about Kustomize because it was announced that Kustomize is going to be part of kubectl. It all felt like this is the way to go. This will be the next standard and it will just be great. But it turned out there's still a lot of stuff that I cannot do with Kustomize. A lot of stuff that gets a lot more complicated, which is a lot simpler when solved with simple templating. And it forced me to do my project layout in a way that did not feel natural to me. Especially because everything required constant context switches for me. Looking at one manifest doesn't give you enough information to figure out what is actually going to be deployed to the cluster. Because there could be overlays involved, there could be patches involved, there could be whatever involved. Requiring you to do a lot of context switches and being aware of a lot of stuff, just for simple things. In my opinion, having a manifest and seeing there is a template allows you to immediately understand what's going to happen without doing any context switches. So, I realized Kustomize is not enough, even though I liked it. And maybe it's important to note that kluctl does not replace Kustomize. It's using Kustomize a lot, same as with Helm. What it does is it builds a project structure on top of those tools.

Bart: Very nice. No, it's always helpful to know how you looked at the things that were available to see. Were you answering a question that was being asked, or were there available solutions that may have already been doing that? So I think it's always healthy to be benchmarking that. In the recent past, we've seen more and more tools trying to tackle the Helm hegemony. Some examples that we might think of could be Kube, Pulumi, CDK8s, and most recently, Apple released PKL and a small European startup that launched called Glasskube. Do you think that any of these have a chance to succeed?

Alex: Obviously, I can't predict the future, and I could be completely wrong with what I think, and I might be the only one thinking this way. So who knows? But I believe that... For example, CUE and jsonnet are trying to solve the same issues while trying to avoid templating as much as possible. And that's kind of what I'm trying to make as a point. Templating is not the enemy. It's not that... It's not the factor that gave us all the pain in the last few years. It was not templating. At least not templating alone. It was a lot of factors being combined. I believe that, for example, CUE... How to explain it? Maybe when I compared HTML to actual programming. So learning CUE is a different thing than learning YAML. And learning YAML in combination with a little bit of templating that does a few text replaces is a completely different thing than really learning a completely new language, for example, CUE. For me, it feels like writing CUE is a completely new thing, and I'm not sure if it's worse. So from the perspective of someone looking at this new solution. I would ask myself, should I really learn a new language with all the ecosystems that come along with it, just to solve this one specific problem, when in the end I just want to see YAML or send YAML to a Kubernetes API? That's what I would ask myself. And I would assume that many other people also would ask that question. But that's what I meant. I could be completely wrong. And we will see what the future brings. I think it's the same with jsonnet. For Pulumi and CDK8S, it's even more special. They actually require you to do programming. And I believe that many potential users are not the same audience that would prefer full-blown programming to deploy some Kubernetes applications.

Bart: If you had to go back in time and save us from YAML and had the chance to rewrite Helm, how would you change Kubernetes resources? Would we be writing INI files?

Alex: With templating? Of course. Saving us from YAML. To be honest, I'm not sure if there's really a requirement to save us from YAML, because YAML is not the evil part here. YAML has some issues that could have been avoided, for example, the issues with different things ending up meaning the same thing, meaning unexpected things. For example, the string NO doesn't mean NO. It means false, stuff like that. There are a lot of blog articles that people can read up on that. And there are other quirks which should have been avoided. So that's maybe something we could save us from if we could go back in time. But I don't think that YAML is the issue.

Bart: And what's the solution going forward? How do you expect the community to evolve around templating Kubernetes resources and distributing them?

Alex: I don't have real expectations. But I hope that people reconsider the topic of templating YAML. And that we don't try to move forward in a way that's like, "Hey, templating is bad, we burned our fingers, let's not touch it ever again," but then have other solutions with other pain points. It would be the wrong motivation to start something completely different. So, as I said, no expectations, but let's reconsider stuff.

Bart: Closing questions. At what point did you decide to write your own Helm, if you want to call it that?

Alex: First thing, as I tried to explain, it's not Helm that I'm trying to replace. You can replace some stuff, but it's not a replacement for Helm. When I wrote these scripts in the beginning, what I tried to explain was that Helm and Kustomize could not solve the actual pain points that I had.

Bart: Another thing is, when talking about open source metrics, gaining traction, getting attention, competing with an established and well-known open source project is very much an uphill battle. How does one prepare for that and keep going?

Alex: I don't know. Keep going is maybe the answer. So, it's an uphill battle. It's important that you're not just doing this. I'm not backed by a company or an organization, so it's me doing it and hoping that at some point in time, I have enough contributors to support it. So, I have some backup. But you're right. It's an uphill battle. And keep going is the solution.

Bart: But sometimes people look just at whether it's stars, whether it's pull requests, whether it's issues, the different things that make that a part of it. And like you said, it's one thing to have an entire company behind you. It's another thing when you're doing it on your own. So much respect to you for doing that. What's next for you? What's on your roadmap?

Alex: In regard to klu CDL, I think we're not much to be honest, because I think it is mostly feature complete. Whatever is coming in the form of features is just small stuff that makes it more complete. I hope that at some point in time, someone will help me with the web UI to improve it, because I'm not a UI developer, and I'm lacking at that point. There's a lot of stuff that could be done to have a powerful UI. And you will see what's next. Okay.

Bart: And if people want to get in touch with you, what's the best way to do it?

Alex: I'm always on the Kubernetes and CNCF Slack. You can find me as codablock. I'm on Twitter, same username. You can find me on LinkedIn with my full name. What else? I'm in some Discord channels, but it's too much so that I couldn't even choose which one to say.

Bart: Like you said, Slack and Twitter would be a good way to go. That's how we got in touch, through Twitter. So, you're accessible, reachable. And once again, really appreciate you sharing your time and knowledge with us today. And look forward to future conversations. So, best of luck to you with Klu.

Alex: Thanks for having me. And thanks for that.

Bart: Pleasure.