Platform engineering: balancing tools and culture in modern infrastructure

Bart: So, can you tell me a little bit about who you are, what's your role, and where you work?

Wojciech: Hi, my name is Wojciech Barczynski. I'm the VP of Engineering at Spacelift and a member of the technical steering committee at OpenTofu. In both of these projects, I work on commercial and open-source versions, focusing on bringing better tools for the DevOps and platform engineering community.

Bart: Now, can you tell me about three emerging Kubernetes tools that you're keeping an eye on?

Wojciech: I'll start with OpenTofu, which provides a strong foundation for your Kubernetes and CNCF initiatives in your company. What's important here is that, as an open-source community project, we've delivered a number of features that make working with infrastructure records easier and more secure.

Two other projects I'm keeping an eye on are OpenTelemetry and OpenFeature. OpenTelemetry is quite popular because changing your observability provider, tracing provider, or monitoring provider can be a hassle and usually results in vendor lock-in. With OpenTelemetry, you can easily move between tools, which is important in today's market where efficiency in infrastructure is key. Having the flexibility to switch providers and improve how we run infrastructure is crucial.

The last project I want to mention is OpenFeature. This project is similar to OpenTelemetry but focuses on feature flags. Feature flags are one of the most powerful tools for software deployment, enabling fast and secure development. We use them extensively at Spacelift, which is why I believe every developer in our community should keep a close eye on this project.

Bart: Now, moving on to our second block of questions, looking at the subject of availability of platform engineering. One of our guests, Hans, compared delivering software over the past 20 years. He mentioned that while downtime was acceptable in the past, it isn't today. Hence, building platforms on top of Kubernetes requires more tooling than ever. Is it possible to keep tool sprawl at bay? And what kinds of tools are essential for building mission-critical platforms?

Wojciech: So, the first part of my answer would be maybe a little standard. You should always start with culture. I think this is actually a setup for a proper mindset, both on the engineering side and operational side. I'm talking here about the culture you build when you're at development time, looking for opportunities to make your software easier to run in production. I think that's a strong weight. Before I jump to platform teams and tools, it's always a good idea to evaluate how much downtime costs you, because reliability is not free. You might not need to have geo-distributed servers. These three elements that I just mentioned are like a base for actually running out your strategy for making your software more reliable.

You mentioned platform teams, and this is a pattern that we see across all of our customers, where we want to create a team that gives a drumbeat for improving operational and development practices across the organization. What's interesting about platform teams, compared to DevOps teams, is that we're also looking for primitives that we can provide to our development team to make adoption of tools easier. Of course, we want people to use as many tools as they want, because we want to have our team autonomous. Thanks to the platform team, we have a mechanism to align them, even if we allow them to experiment with different tools. We have one place where we can make decisions about which tools to use.

Because of this, if your company is a little bit smaller, I would recommend creating a DevOps platform engineering field that could drive this kind of initiative.

Bart: Now, you mentioned GitOps and looking at infrastructure as code, those two worlds coming together. One of our guests, Dan Garfield, said that using Kubernetes as a central data source allows tools like Argo CD to detect and sync drift in your infrastructure. In comparison, tools like Terraform externalize their state and are harder to track. Is the market moving toward tools like Crossplane to provision infrastructure and away from Terraform?

Wojciech: So what I see currently is that Crossplane is quite lightly adopted, mainly in very innovative projects. I see it as an important milestone in the direction of autonomous infrastructure in the future, where we'll have a magic mechanism that fixes our infrastructure, but I don't think we are yet there. Regarding Terraform or OpenTofu, you can achieve similar results if you're using something like Atlantis and Spacelift, where you can continuously evaluate OpenTofu and Terraform to detect changes and react on them. If you combine this with policies, you can also make automatic decisions on whether a detected change should be overwritten or if a human should review it before making a decision. Regarding OpenTofu, we are looking for new ideas to address the problem of managed and unmanaged drift. We are also looking for contributors and ideas to bring to the project to make this problem simpler for others.

Bart: Now, Kubernetes turned 10 years old this year. What should we expect in the next 10 years to come?

Wojciech: I would expect Kubernetes to get a bit simpler to operate, especially in terms of scalability, going up and down the cluster, and operational aspects in general. Another thing, which is more of a wish than a prediction, is that I would love to see fewer changes in the upcoming years. I think Kubernetes is in the right place, so we can make changes at a slower pace. Since Kubernetes is deployed in many companies and people rely on it, adding more features might make operations complex for many companies to run. I have experience with OpenStack, a technology with a very active community. In the early days of OpenStack, there was a period of rapid feature addition, which made it hard for companies to migrate. This was probably eight to ten years ago. I wouldn't want to see similar effects on Kubernetes. I don't think we're at that risk, though, because the Kubernetes project is well-maintained and run. It's just a small wish to slow down the number of changes.

Bart: I think there are a lot of people who would agree about that part of complexity. It's something that comes up in a lot of interviews we did. As a follow-up to this, we spoke to Bob Lai, CEO of Heroku, on the last KubeFM. He mentioned that the biggest challenges with Kubernetes are not the technical ones, but rather the human ones. I wanted to go back to your response to one of the previous questions about organizational culture and mindset. When it comes to Kubernetes, would you agree with Bob that the biggest problems we have are getting people in the right place when it comes to working with Kubernetes, as opposed to just the technical challenges that might surround it?

Wojciech: I would agree. Kubernetes requires a different way of thinking about applications. We can see this in the community, for example, with the rise of Argo CD, which delivers a graphical visual representation of the cluster state to development teams. This makes it easier to follow changes and identify what's going wrong.

Bart: In terms of what's next for you as a person and also for Spacelift, tell me about that.

Wojciech: For Spacelift, we are looking to increase our commitment to our platform. We see that we are gaining momentum in the project, with many big enterprises already moving to OpenTofu. We want to keep this momentum going by continuing our support for the open project. We plan to bring more core developers on board to maintain the momentum, as more and more companies are supporting the project and integrating OpenTofu. OpenTofu has appeared in the documentation of most major cloud providers, and Cisco is treating OpenTofu as a first-class citizen in their ecosystem. We want to keep this momentum rolling, and this is the next important thing for our company. As part of that, Spacelift will continue to grow and help DevOps and platform teams effectively and efficiently use infrastructure, ultimately serving their primary customer - the developer - so they can move faster and more safely.

Bart: If people want to get in touch with you, what's the best way to do that?

Wojciech: The best way to reach me is by LinkedIn. You can find my LinkedIn profile by typing my name and surname. I'm not going to spell it here because it would take a lot of time. You can probably find it as the first or second result. You can also reach me by email, and I'll share the notes for this recording. I work for Spacelift.