From infrastructure tools to platform engineering: the evolution of Kubernetes

Bart: Who are you? What's your role? And where do you work?

Asif: I'm Asif Awan. I am the co-founder and Chief Product Officer at StackGen.

Bart: What are three Kubernetes emerging tools that you're keeping an eye on?

Asif: We are definitely keeping an eye on Crossplane, Argo Rollout, Argo CD, as well as KubeCost.

Bart: Any reason in particular for those three?

Asif: Because StackGen operates in the DevOps infrastructure as Code space, we are definitely keeping an eye on Crossplane very closely. They're right here.

Bart: These are reactions to episodes that we recorded on our podcast. Regarding the topic of automation and resource management, Alexandre expressed that having an automated mechanism is better than enforcing processes. What automation tools or approaches do you recommend for managing Kubernetes resources?

Asif: I have a slightly different perspective on automation. When you talk about automation, I think it's also important to consider abstraction. What I mean by that is that when the focus is on processes, those processes are often specific to certain enterprises and the way various groups within the organization have been structured. For me, automation needs to take care of that level of abstraction so that users do not have to learn anything new, such as a new language or tool, and then provide them automation at a level of abstraction that they can understand and benefit from without the additional burden of learning new tools.

Bart: On the subject of GitOps and infrastructure as code, Dan Garfield said that using Kubernetes as a central data store allows tools like Argo to detect and sync Drift Detection in your infrastructure. In comparison, tools like Terraform externalize their state and are harder to track. Is the market moving to tools like Crossplane to provision infrastructure and away from Terraform?

Asif: I think the market is moving away from Terraform for a different reason. It's because of the complexity involved in having to deal with Terraform on a day-to-day basis and managing it. It's not so much from the Drift Detection perspective, because I think there's a reason why Drift Detection is introduced in the cloud. The reason for this is that SREs, who are responsible for keeping the lights up and business applications running, make changes to cloud resources directly. So, blowing away those changes by overriding them using a Crossplane or Kubernetes approach may not be the right thing, as those changes are necessary. What needs to happen is to automatically figure out the Drift Detection and provide a way to easily reconcile it. Two key points are the reconciliation of Drift Detection, or closing the loop, and dealing with the complexity of Terraform, which is a separate problem. This needs to be addressed by having an abstraction layer, so day-to-day developers and DevOps engineers do not have to worry about dealing with Terraform.

Bart: When we're talking about platform engineering and people, one of our guests, Ori, shared that rushing into solutions without understanding the root cause can lead to fixing symptoms instead of the actual problem. He mentioned the case of Network Policies and how sometimes the root cause of a problem is a people problem, and that the solution lies in addressing that. What is your experience with providing tooling and platforms on Kubernetes to other engineers? What are some of the soft challenges that you have faced?

Asif: I actually totally agree with the comment. Based on my experience, we often make two mistakes. We jump to building or providing solutions rather than understanding the root cause. This is a key point, as he rightly mentioned, that we need to focus on the root cause of the problem. The second thing is that when it's a people problem, the assumption is often made that Kubernetes is complex. The typical assumption made by those responsible for building the solution is that everybody has the same level of knowledge or expertise, which is not true. In my past experience, I have found it essential to understand the root cause of the problem and focus on documentation and workshops to provide capabilities and address the knowledge gaps of different user personas. It's not a valid assumption that everybody has the same level of knowledge, and then assuming the solution provided will successfully address those pain points.

Bart: Kubernetes turned 10 years old this year. What should we expect in the next 10 years to come?

Asif: I think it's going to continue to grow very rapidly from here onwards. The last 10 years, if they are any indication, I'm expecting massive growth and a massive expansion of the ecosystem as well. It is already a de facto standard for the container orchestration layer, but going forward, it's going to be the main compute platform for deploying application workloads.

Bart: With that in mind, I believe if I'm correct, the last time you attended KubeCon was in 2017. Can you tell me the growth and differences you've seen from seven years ago until now? As you just mentioned, the community, all the things in terms of adoption, what's the difference between then and now?

Asif: Just the ecosystem. Like I said, it has become the default platform. It's interesting that in those days, we were evaluating whether it would be Kubernetes, Mesos, or Docker Swarm. But now, obviously, terms like "Kubernetes-native application" are rarely used; nobody talks about "container-native" anymore. It's just assumed that containers will be deployed on Kubernetes or using the Kubernetes orchestration layer. This is a huge thing - it has become incredibly popular and common. At StackGen, we're experiencing very good traction with customers, and what we've built is resonating well with them. We've already signed a lot of large enterprise customers and have a strong pipeline, so we're looking forward to a rapid growth phase in the coming year. For me personally, this is what I like doing - this is my fourth startup. I just keep doing this, but this particular journey is the one I'd like to take all the way.

Bart: How can people get in touch with you?

Asif: People can get in touch with us, Bluesky, or X at StackGen, and with me directly on LinkedIn.