When Kubernetes and Go don't work well together

Bart: What happens when Kubernetes and Go don't cooperate? In this episode of KubeFM, I got a chance to speak to Emin, a solutions architect at Hybirid Technologies. He'll share his experience when he faced an issue - a 502-bad-gateway error that appeared only in a specific production environment, which was then traced to an out-of-memory or OOM issue. In this case, Go's garbage collector wasn't aware of Kubernetes resource limits, causing excessive memory allocation. The logs showed that Go's heap size kept growing until it exceeded the container's memory limits. To find out what happened next, you'll have to stick around for the full episode. You'll learn how debugging Kubernetes issues often requires a systematic approach and that understanding resource limits is crucial in Kubernetes environments.

This episode is brought to you by StormForge OptimizeLive. StormForge Optimize Live continuously right-sizes Kubernetes workloads to ensure applications are cost-effective and performant while moving developer toil. As a vertical rightsizing solution, OptimizeLive is autonomous, tunable, and works seamlessly with the Kubernetes HPA at scale. Free developers from the toil and complexity of setting CPU resources and memory requests with OptimizeLive's fully configurable, machine-learning-based recommendations and intelligent automation. Start a free trial or play around in the sandbox environment with no form fill required. Now, let's check out the episode.

All right, Emin. What three emerging Kubernetes tools are you keeping an eye on?

Emin: Emin, I'm not really up to date with new emerging tools. I stick to the greatest hits and things I use in my daily work, like kubectl, Prometheus, Grafana, and Minikube.

Bart: Now, what do you do and who do you work for?

Emin: I'm a software developer and solution architect at the Ministry of Programming, a startup studio from Herzegovina, that helps its clients create new products, iterate, and succeed faster. I'm currently assigned as a solution architect at Hybirid Technologies, an innovative asset management company in the oil and gas industry.

Bart: And how did you get into cloud native?

Emin: My journey started when I joined Ministry of Programming. Their technical direction at the time, and still is, to use cloud-native tools and frameworks and technologies as much as possible, unless clients require some other setup. That was basically when I started working at MoP.

Bart: And what were you before Cloud Native?

Emin: At my previous company, we mostly used Microsoft-related technologies and frameworks without any, or very few, cloud-native options. Most of the products we worked on were in the mortgage industry, and they weren't very keen on using cloud-native tools back then.

Bart: And the Kubernetes ecosystem moves very quickly. How do you stay up to date with all the changes that are happening?

Emin: I can't say I'm up to date with everything in Kubernetes and our native ecosystem. I always like to describe myself as a software engineer and solution architect who prioritizes breadth over depth, but I can double down on specifics if needed. One thing I like to do is subscribe to various newsletters to stay informed and keep in touch with everything that's going on in specific areas. I also really love talking to my colleagues who hold more depth in this area and learning as much as I can from them.

Bart: If you could go back in time and share one career tip with your younger self, what would it be?

Emin: I would say, spend less time watching TV shows and more time on books, courses, and articles. It hit me really fast, and unfortunately, I haven't learned the value of my free time until after I had kids. From a more technical perspective, I try to learn and try out as many different things as I can, rather than being a master of a few. This helps to keep my horizons broad, gives me more opportunities, and provides more solution possibilities, rather than limiting myself.

Bart: Now, for our podcast today, we found an article that you wrote with our monthly content discovery, "When Kubernetes and Go don't work well together". Let's start with the issue that you encountered. You mentioned a specific API endpoint was failing in one of your production environments. Can you walk us through what happened and how this problem first came to your attention?

Emin: I received a report that our REST API service in one of the client's production environments is returning a 502-bad-gateway error for a specific endpoint. We checked other environments to see if the issue manifests everywhere. Since all other services, specifically that service in other environments, worked just fine, we isolated the issue to the specific environment and started investigating in more detail. We went through the usual things, checked the logs, but found nothing. However, one thing we could clearly see when we called the endpoint is that the Kubernetes pod restarts every time the endpoint is called.

Bart: Now, before we dive too deep into the technical details, it would probably be helpful to understand more about your setup. Can you give us an overview of your production environment and the type of application you're running?

Emin: Our production setup for this REST API service, its container, and pod has 20 millicores for the CPU and 400 megabytes of RAM, which are the Kubernetes resource limits. Although we have other services in our architecture setup, this service is the most used one and contains all the project data. It's also worth mentioning that the problematic endpoint works differently than the other endpoints. It provides data for the offline mode of the mobile app, which means it returns all the data for the bank project.

Bart: With that context in mind, I suppose we should return to the issue at hand. Debugging problems like this can be challenging. What were your initial steps in investigating this mysterious error?

Emin: When we isolated the problem to that one environment, I said we'd check the logs to see what happened. The odd thing was that there weren't any occasional, panicking, or critical errors. So the next step was to add more debug logs to determine where the service stopped working. We ended up learning that it stopped just after the json.Marshal function was called. The main concern was why it stopped working, especially since the data from the database returned to the service successfully, and that function works everywhere else. There were some questions to be answered.

Bart: Now, as you dug deeper into the pod's behavior, you mentioned discovering something interesting about its status. What did you find, and why was it surprising given what you knew about the application?

Emin: After determining where the service errored, we checked pods and container-related information to gain more insight. We ran kubectl describe pods and found the OOMKilled status, which means the container went out of memory and was terminated for that specific reason. We wanted to test this out, so we scaled up resources to 400 CPU resources and 800 megabytes of Kubernetes resource limits, triggered the endpoint again, and it stopped working properly. However, the endpoint returned a response of 66 megabytes, which was way below our 800 megabytes limit.

Bart: OOM killed status is a critical issue in Kubernetes. I think it's a known fact, but it might not be familiar to all of our listeners who perhaps don't have experience with it. Could you explain when OOM occurs in Kubernetes and how it relates to the Kubernetes resource limits you've set, specifically in relation to container memory limits.

Emin: When a process running inside a Kubernetes pod tries to allocate more RAM than its defined container Kubernetes resource limits, an OOMKilled error occurs to prevent misuse. At that point, the daemon restarts the container and the pod to prevent further memory allocation. In the context of our REST API service setup, if the service tries to allocate more than 400 megabytes of RAM, the container and pod are terminated to prevent that. If we refer back to our findings, the question remains the same: how can a 66-megabyte response cause an out-of-memory error for a pod with an 800-megabyte limit?

Bart: This seems to point to an interesting interaction between Go and Kubernetes. Based on your investigation, what did you discover about how Go behaves in this containerized environment?

Emin: So, after some research, the answer pointed to two things: how Go's garbage collector works and how Go processes work in a containerized environment.

First, a short introduction to the garbage collector. In short, the garbage collector checks if there's some heap memory that can be freed up and frees memory, among other things, and runs cycles. Before each cycle, it determines if more heap memory needs to be allocated to that process or not. If the answer is yes, it allocates more RAM, specifically doubling the amount of RAM previously used.

The other part of the answer is how Go works in a containerized environment. When a Go process is running inside a container, it doesn't know about the resource limits set on the container. It's only aware of the limits of the machine that the container is running on. For example, if we deploy a container with a memory limit of 400 megabytes to a machine that has 8 gigabytes of RAM, the Go process knows only about the 8 gigabytes of RAM. And when the garbage collector runs and expands memory allocation, it will not stop at the container's memory limit.

Bart: That's a fascinating insight into how language runtimes can interact with containerized environments. How did you go about confirming that this was indeed part of what was happening in your case?

Emin: At that point, this was just theory based on research. To confirm this theory, we changed the GODEBUG environment variable to include trace logs for the Go's garbage collector. With this in place, we were able to see how the Go's garbage collector works and that, just before the process was terminated, it was able to allocate 260 megabytes of RAM. So that means that, before the next run, the Go's garbage collector tried to allocate more than 500 megabytes of RAM, which is more than the 400-megabyte limit, and the container system terminated the container and the pod due to being OOMKilled, which is related to Kubernetes resource limits.

Bart: Now that you had identified the root cause, it was time to find a solution. What approach did you take to address this mismatch between Go's garbage collector and the container's Kubernetes resource limits, specifically the memory limits that lead to OOMKilled?

Emin: One thing that was added with Go 1.19 is the GOMEMLIMIT environment variable. This environment variable is used as a soft limit for a Go process running on a specific machine. By default, this environment variable is not set, but when it is set to a specific value, the Go's garbage collector works more aggressively to try to free up more memory and keep the memory below the set value. This approach worked for us because, when approaching the limit, the garbage collector doesn't double the allocated memory anymore but runs more efficiently and uses that memory more effectively. In our example, when we set this environment variable, the garbage collector didn't try to allocate more than 400 megabytes of RAM, and we got a 66-megabyte response when triggering the endpoint.

Bart: It's interesting that this requires manual configuration. Why do you think this isn't automatically handled? And what are the implications of needing to set GOMEMLIMIT ourselves?

Emin: I think that it stems from the fact that Go hides the inherent platform behavior. This one is even more important, that by default, all the default settings work best for most instances. Sure, there are always trade-offs. We can set the command element to a specific value, but then you will have the scenario that the Go's garbage collector works more often and more aggressively, and it takes away runtime from your service.

Bart: Now that you've implemented the solution, how effective was it in resolving the issue? Are there any potential drawbacks or limitations to this approach?

Emin: Even this work was a temporary fix for us. Long-term, we had to refactor the problematic endpoint, move some memory allocations from the heap to the stack, and implement paging for the input. So, I would say a less working boot really big memory allocation to always use the default settings and use code very carefully.

Bart: This experience seems to highlight some important considerations for developers working with Go and Kubernetes. What do you see as the key takeaways from this incident?

Emin: One key thing I would take from this experience is how much language around time and direction is important. When you try to tune your services to work in the right environment, it's best possible. As always, this is not needed most of the time, but when you want your mission-critical services to perform well, it's really a must. Similar scenarios to this would involve utilizing CPU resources more optimally. For example, in an environment like a Kubernetes cluster, your workloads are deployed together with Kubernetes-related workloads that help with running your Kubernetes cluster more smoothly on one physical machine. All of those workloads fight for the same CPU resources of the cluster nodes. If your setup doesn't involve a careful plan for which workloads are deployed where, and you don't set up runtime limitations on how much CPU can be used by each workload or set CPU affinities, you will probably not be able to have predictable performance outcomes.

Bart: Now, it seems that your experience could challenge some common assumptions about containerization. How does this case study reflect the reality of running applications in containerized environments, particularly when it comes to the idea that containers provide a universal solution?

Emin: So, it's time for everyone's favorite answer: it depends. In the majority of the cases, you will be totally fine. In all other cases, as few as they are, they involve going deep into the internals of language runtime and containerized environments and understanding how they work. Or, in some cases, as some studies and case studies point out, ditching the containerized environment. This really depends on the business and technical requirements, costs, and benefits of pursuing the solution to that extent. I'd like to paraphrase Mark Richards, who says that most of the time, you're not going for the best possible solution, but the least worst one.

Bart: Great way of putting it. This has been an incredibly insightful discussion. For our listeners who are out there who might be facing similar issues or that might want to dive deeper into this topic, are there any resources or final pieces of advice that you'd like to share?

Emin: There are great Ardan Labs articles about optimizing your Go programs and workloads, both for memory and CPU consumption. The Weaviate blog post on how GOMEMLIMIT helped them out, and also point out the official documentation on Go's garbage collector and how it was extensive and a fun read.

Bart: Now, what's next for you?

Emin: I'll continue to learn and explore, as always. I'm very interested in all things related to performance improvements, so that is and will be my focus. For now, I don't write very often, and I usually write about my experiences and interesting topics that are less covered and hard to find on the internet. If you don't hear from me in a while, it means either I don't have time to write or there weren't any interesting production issues to share.

Bart: It's not necessarily a bad thing.

Emin: I'm ready to assist. However, I don't see any text to process. Please provide the transcript text, and I'll be happy to help you identify words or terms that could be explored further and add hyperlinks to other resources according to the guidelines.

Bart: If people want to continue the conversation and get in touch with you, what's the best way to do so?

Emin: I have a couple of blogs, a Hashnode and Medium blog, and I'm also on LinkedIn. Feel free to say hi, ask questions, share your concerns or different opinions. I'm happy to hear from anyone.

Bart: Very good. We're happy to hear from you. It's easy to reach out and connect. Thank you for sharing your time and knowledge today. This is a topic that hasn't necessarily come up in other episodes, so I'm sure our audience will be very keen to hear from you and also to read about the other work that you're doing. Emin, thank you very much for joining us today.

Emin: Thank you for having me. This was great.

Bart: Thank you. Pleasure. Cheers.