The Making of Flux: The Rewrite

Bart (Host): In episode 1, we traced Flux's origin story with Alexis Richardson, Chris Aniszczyk, and Andrew Martin, moving the project forward and steering it to CNCF graduation. In this episode, we rewind to its very first commit and then fast forward to the operator era. Michael Bridgen, the engineer who typed Flux's first lines, drops us into 2016 when Kubernetes upgrades meant SSHing into nodes and praying. Stefan Prodan, the maintainer who led the V2 rewrite, explains how Flux became the GitOps workhorse teams rely on today. By the end of the episode, you'll see how a hackday script turned into the toolkit behind everything from CI pipelines to air gap factories and where it's headed next. Michael Bridgen (Original Engineer): Well, at Weaveworks, that was the company where Flux originated. We were earlyish adopters of Kubernetes. So not the earliest but you know 1 point single digits certainly and lots of things were still kind of being settled on. So for instance, CNI was around that time still being standardized and there was a lot missing still. So the problem we were trying to solve was being able to safely roll out new services or new versions of services and this is when there wasn't even such a thing as a deployment and there weren't CRDs anything like that, or service meshes for that matter. So our first run at this was we had a proxy that ran in each pod and it could be programmed to route traffic and we did sort of AB rollouts or blue green rollouts that way. That was wildly ambitious for where we were at the time. So what we ended up doing was we just wrote a pretty simple daemon which you would tell it what image you wanted to update to and then it would progressively roll that out using the means that we had at the time which was replication controllers. And then it would write the result of that back to Git because we had this quite strong culture of practice of keeping everything we could, all the config which at that time included lots of things like Ansible but also Kubernetes manifests in Git. So we wrote the result back to Git when it succeeded. And that was quite successful and from there it was very short step to saying well since we've got everything in Git anyway why don't we write to Git first and then just apply whatever it is and that will work for more than just rolling out images. That'll work for everything if we want to make any change. And that's probably what I would say was the first thing that was recognizably Flux. Bart: Back in episode 1, Alexis called Flux automation that remembers intent. Michael just showed us the moment that intent crystallized. But a good idea only matters if the community shows up. Michael Bridgen: Yeah. So we were our own customer and our own sole customer for a while. We eventually developed part of the product, the Weaveworks product, to interface with Flux so that you could do sort of push button deployments which was quite popular with some people. Some people really loved it and for a long time loved it and even were reluctant to move off it. But I think the thing that really changed Flux to be a sort of more upward trajectory in terms of adoption was the Helm operator as it was then because that seemed to click with a lot of people who had trouble reconciling Kubernetes model of doing stuff with how Helm worked which was very imperative, whereas the Helm operator let them say you know this is what I want to be the case which was much more of sort of natural Kubernetes thing. Not everyone but lots of people kind of clicked with that and I think probably for a lot of people it was the Helm operator that they really wanted and the Git part of Flux was actually by the by, that was just a sort of incidental detail. And that was the first time we ever experimented with CRDs. They were like alpha back then. Yeah, that's right. That's when we made the Helm release custom resource. So it was like early days of Kubernetes CRDs, very very anti-extensible. That's right. That's why it was a Helm operator because it did actually rely on CRDs whereas Flux, the rest of it, none of the CRDs, none of that stuff was around. No controller runtime, no controller runtime client which made a lot of things easier later on. We had to rely on client-go. That was very tricky. Yeah, you're right. That was the first kind of controller. Stefan Prodan (Flux V2 Lead): Yeah. So we went to I think it was the second Helm summit. It was in Amsterdam. And we went there. We showed the world our Helm operator. And I think that was the moment where things actually took off where we actually demonstrated we can apply this GitOps idea that was also something that you know it wasn't popular like today. Most people didn't know what GitOps is. It just sounds funny but you don't exactly know it. And when we showed that we can apply the GitOps principles to Helm, that was the moment where the community started to you know exponentially grow from that point in time I think. Bart: So early adopters filed PRs faster than Michael could merge them. That momentum caught Chris Aniszczyk's eye, the sandbox invite we heard last time. Here's how Flux handled the CNCF gauntlet and doubled its feature set. Stefan joins the timeline. Michael Bridgen: Yeah, I can talk a bit about Sandbox. Stefan was much more involved and hands-on with the graduation part of it, but for Sandbox that was sort of a decision in two parts that was actually made by Alexis. One part was that it was good for Weaveworks because there's a sort of reflected glow of the CNCF onto Weaveworks and the works of Weaveworks. But the second part which actually long-term was a more important thing was making Flux into a project that was independent in terms of governance, you know decisions around features and so on of Weaveworks or indeed you know a vendor. That was quite a brave thing to do I think, maybe less brave, only brave with hindsight or more brave with hindsight. But you know there were times that we said no to things that Weaveworks wanted because there were you know people in the community that wanted it to work a different way or wanted something more important. So that was quite a break. But it did, I think going into the CNCF did benefit Flux because it did kind of give that home for it that made it not directly associated with Weaveworks. It's kind of its own thing and it could accumulate people that might have been either users or contributors that might have been wary otherwise. Stefan Prodan: We were also early, CNCF was also early on there. Yes, there weren't like strict rules or there were some rules in place but there wasn't like a definitive way of how you go from sandbox to graduation. They were changing. I think it was the first TOC, technical oversight committee, was like the first iteration of that. Yes, when we got sandbox status. We could have tried to get into incubation first but I think this was mainly a decision between us the maintainers like let's go and do the whole trip basically. Let's start from sandbox and we'll work our way to graduation. It was very long. We spent a long time. Yeah. So we got into sandbox in 2019 and graduation was end of 2022, almost 2023. We almost spent four years following the road to graduation. And we learned that open source is more than just you know opening pull requests and talking to your users. It involves you know collaborating with other projects, being involved into KubeCon activities, all the things around CNCF TAG. We interacted a lot with TAG Security and other organizations inside CNCF that helped us mature Flux or at least give us some guidance on how to do it. And that was I think the real benefit of being into CNCF. It wasn't only us figuring out how to secure Flux, how to create release pipelines, how to write our docs. We reached out to CNCF in various ways and they helped us. For example the Flux CD website, it was someone from CNCF that made it. We weren't web designers, we didn't know how to write CSS and all of that stuff. And we got help in various ways. I would say from sandbox to incubation that period we accumulated new users, the community was there. But I think the moment when we got into incubation, and we'll talk about it, the rewrite of Flux and Flux version two which was at the time of when we got the incubation, that's when you know it actually took off in various directions in terms of adoption. Also adoption in enterprise, adoption in like people were running Flux for home automation. So yeah I think if I'm looking back I don't regret anything around interacting with CNCF and I think it was worth it going through all the stages and we learned more than you know how to build software, also learned how to build a community, how to collaborate with other projects along the way. Michael Bridgen: Yeah, and I think as you are suggesting going through all of the stages meant that we kind of learned all of the things that you know it wasn't a checkbox exercise and we got sort of the benefit of having to prove ourselves for incubation and then the same thing for graduation even more so. And that actually involved you know material differences, changes to how we did things or you know I was less hands-on then but you know the security audit probably changed quite a lot of how certain things were thought about. Stefan Prodan: Oh yeah. Yeah. We went through two security audits which were sponsored by CNCF. I mean they helped us. We needed to fix things after the first audit and we actually changed, we made architectural changes after the first audit to build a more secure Flux for the future. Let's say we kind of redesigned a little bit how we interact with the Kubernetes API. Bart: Sandbox status was nice. Production grade was mandatory. Flux v1 hit scaling limits just as GitOps took off. Stefan led the great rewrite, v2. Here's what changed and why. Stefan Prodan: So we started V2 as an experiment on the weekend. It was me. We were asking Michael all the time, what do you think about this? What do you think about this idea? And yeah, after V2 became a thing and we've seen people using it in various ways that you couldn't use Flux V1, we actually decided, you know, we are going to create a transition roadmap for Flux V1 users to be able to migrate to Flux V2. We didn't want to disrupt the community. We wanted to move everyone to the new version. So with Flux version 2 we actually kept the idea and the principles of GitOps but the source code, the design, everything was new. We rewrote it. It was a full rewrite. And when you do this type of evolution of a project you should really care about your current users. We needed a good plan for them to transition from one version to another. And we were as a project very fortunate that Weaveworks employed great people back then to help us, to be on the Flux team then and you know create a plan on how we are going to migrate the V1 users to V2. It took almost one year. It was a big effort to do this migration. And yeah we didn't end up with a community that was split between the two versions. After one year everybody that we know actually managed to migrate. Michael Bridgen: And we deliberately gave ourselves along that year and you know users a year in which to get from V1 to V2. That was both for us and for them wasn't it? Stefan Prodan: Yes. And we set up a goal with V2 like we need to have feature parity even if you know the APIs are different, everything looks different, but in terms of features we should have feature parity between version one and version two. And once we hit that milestone then we started this migration path for our users because we knew whatever they are doing we have the answer for them. We have a solution for them to migrate to the new shiny APIs. Flux V1 was built before CRDs were a thing. In order to create this new idea of Flux which is Kubernetes native, built on controller runtime, it uses, it's as close as possible to how Kubernetes controllers are, we needed this total rewrite. And we basically wrote a translator between Flux V1 flags, how you started the controller, to custom resources. We have the Git repository custom resource and we know what parts from here to take to generate the custom resource and we had this kind of migration tool back then and I think it was successful. We didn't have bad feedback from users that have migrated. It was like really really great experience. We were I think almost five people worked on this for one year going on meetings with on calls with users not customers. So we weren't doing this and getting paid. We were doing this to you know move the project forward and we realized we can't leave our early adopters behind and move with the new shiny thing ahead. Bart: So Alexis called V2 a quantum leap. Andrew called it pager friendly. Successes, yes, but what would Michael and Stefan redo if they had a time machine? Stefan Prodan: We hit a crisis so to say where we weren't able to advance the project. Even if we had contributors to Flux V1, everybody wanted to change something and we were at some point afraid that if we merge this contribution we weren't able then to merge another contribution. So there were like two pull requests in parallel, they were touching the same code and we realized we will definitely have conflicts here. So we have to pick one and we were in this kind of situation where we realized we need to rearchitect the thing to enable better contributions so people could add features to Flux that wouldn't disrupt other things in the codebase. And for Flux version two, how we decided what things we want to add, we looked at issues that were opened on Flux V1 that had like many votes. Everybody wanted it and we couldn't deliver it in V1. So when we designed version two we already had this backlog of highly requested features. Everybody wanted special things like notifications for example or smarter garbage collection and all these things. And I think that was the advantage. That's why we managed to move so fast with version two and in one year move everybody in production on the new version because we had this backlog. We knew what people would want from the new version. Bart: While they were refactoring, Argo CD proposed a joint GitOps engine. Remember Andy's tale of friendly rivalry. Michael explains the near merger. Michael Bridgen: So that's right. The idea I think in part came from Alexis and the folks at Intuit who ran Argo. The idea, you know quite simply, was we've got two things that are kind of occupying the same space. Why do we not make one thing that kind of does all of it and combine efforts that way? And what we realized was that Flux, this is Flux V1 still, Flux V1 was quite monolithic and quite opinionated about what it did. Argo CD also in a sense quite, at least at the time, sort of monolithic. It's quite all or nothing or it was. And so there was a sense in which these things were two monoliths sort of running parallel, not quite parallel tracks. And when we tried to actually affect this change of working on one thing, it really felt like the will wasn't there from the engineers who perhaps had different outlooks. Which I think you know Flux and this is still kind of true, Flux has the outlook of being very sort of I guess command line driven or you know system daemon kind of thing, hands-off automation. Whereas Argo is ultimately you know there's lots of stuff in there for end users like a very good UI where you can see things happening which Flux has never really attempted to do in the same way. And so it felt like we were on not quite parallel tracks and it wasn't really, it didn't quite work to merge the things in the end. So it sort of didn't happen in practice and then it officially didn't happen. Bart: So the project stayed separate. Was that the right call? Michael weighs in. Stefan Prodan: We don't assume people make a change then they stare at dashboards to see what's happening. You should, I think pipelines and especially continuous delivery should be something that you can fully automate and forget about it and it should work. If something fails then the system is responsible to tell you that something failed. You shouldn't be observing it all the time. So I think it's also very obvious in the design of the two projects that there are like different principles when we started the thing. And you ask like why, if I'm happy that Argo and Flux didn't merge. I think in open source you should always have options right. It shouldn't be like one winner or only one project that can do, is you know monopoly. Don't have monopoly on a particular technology. And I think if people want to do GitOps they should have more than one choice and if we would have merged back then early on we'd have been the GitOps solution in CNCF and I think it's great to have at least two. So I can say now I'm working at Control Plane, we build this enterprise edition of Flux. So I'm interacting more with you know large organizations that are having hundreds and hundreds of clusters, they have many dev teams, many people are collaborating on a deployment and so on. And asking them why they went with Flux, the answer was in part around you know resource consumption and how easy it is to run it at scale but it was also the part where products can be integrated into their existing platforms. We designed Flux in a way that like Lego pieces, you don't get the deployment platform is the component where you build on top of it and you make your own thing out of it. Bart: Features are great but users vote with clusters. Stefan breaks down why teams still pick Flux when audit trails trump UI policy. Stefan Prodan: I think large organizations, enterprises, they are all unique in their own ways of doing things. So Flux is a better fit from that perspective. If we also look at GitLab, they had to take this decision. The amount of due diligence they did on that was incredible and it's public. You can actually, it's public. Yeah. You can see. So they compared like okay we want to give GitLab users a continuous delivery tool which is based on GitOps principles and they had to choose between Flux and Argo. And their answer was more a direction that we can integrate more with it and we can build our own things on top of it, is more flexible. So another advantage of Flux and what we usually hear from users is that Flux Helm controller does Helm like Helm was designed to be used. We use the Helm SDK, we collaborate with the Helm maintainers and so on. So if you are a Helm user, you'll probably like Flux more than any other tool out there because, you know, it behaves in the same way, but you can run it at scale. You can apply, you know, GitOps principles to delivering apps with Helm. And yeah, we have users deploying thousands, tens of thousands of apps with Helm across hundreds of clusters and Flux works great with it without having to allocate or spend money on the continuous delivery tooling. You spend money on running the apps, not the tool that deploys it. And I think that was great achievement so far of the project. Bart: Enterprise users wanted support without a walled garden. Control Plane's answer: enterprise Flux. Stefan Prodan: Yeah. So we started building Flux Enterprise roughly one year and two months ago. And the goal with Flux Enterprise was finding a solution to sustain the Flux project and be able to have people working on Flux full-time. So that was our goal. How can we achieve that goal? You need to offer something that gives enterprises or large corporations assurances. And Flux Enterprise is about compliance, is about being able to you know have 24/7 support. Whatever happens with Flux we are here to help you and you get the help from the maintainers, from the best people that know how Flux works. It's also about the fact that Enterprise is not just you know we ship a hardened version, it's also us that are doing it. So we can also you know deliver guidance, architectural reviews. And with Enterprise for example we publish, they are free, everybody can read them and exercise them. We publish this architectural designs where we show enterprises how Flux can be run at scale, how we can secure it. Bart: Day 2 chaos as Andy put it meets its match: the Flux operator. Stefan Prodan: Oh yeah. Flux operator is a new shiny thing we are building at Control Plane. The main, how it started, the main driver was we know that our large enterprise users, some of them are using OpenShift and Flux never had good integration with the OpenShift operator hub and all the things how you do it in OpenShift, how you deploy operators there. They have their own SDKs, everything. So it was that need to create something that is fully integrated into the OpenShift ecosystem. So that's how Flux operator started. It was a way of deploying Flux natively into OpenShift. Then after doing that other Flux users were deploying Flux with the Flux operator at scale and they were not running on OpenShift. So I kind of saw the idea of okay this is the place where we can be more strategic around and be more opinionated around how you configure Flux at scale, how you secure it. And we've built this let's say user experience improvement into managing Flux at scale, being able to upgrade Flux, always on the latest version. Bart: We've covered the past and present. Now, here's the roadmap for the future. Stefan Prodan: If you, for example, Flux multi-tenancy, where you secure the delivery of an app and whatever thing controls the app, for example, they can't change things inside the cluster outside of the scope of their app. If the app is limited to a namespace, it only needs to talk to three APIs and so on, then you have a means in Flux to configure that and Flux operator makes this very simple, straightforward and it protects you from doing mistakes when configuring Flux. So that's one part of how the operator started. Now we are at the phase where we are building these high level APIs on top of the Flux APIs. And an example here is that we wanted to allow people to have this self-service mechanism where, one example is, you open up a pull request, you change the code of your application, maybe you change the configuration of that application. Let's say you make a code change, you add a new feature, then you also go into the application Helm chart and there you add configuration for that particular feature. Now with plain Flux, how would you deploy that? You'll have to merge your pull request, get the configuration in a branch, then Flux running on let's say the dev cluster will sync those changes there. If you made a mistake in the configuration or in the app itself, you have to open another pull request, you need to fix that. But you didn't have a means to test that change before it got merged on an actual cluster. So Flux operator integrates with GitHub and GitLab for now with their pull request and merge request APIs and can dynamically spin up these ephemeral clusters, ephemeral environments in test clusters where you don't have to merge things. You can actually from the pull request by adding a label, let's say Flux preview, you can deploy the whole thing in a test cluster. And after the pull request is merged Flux operator will go and will clean up all the things that were deployed without the developer worrying about the Kubernetes environment and so on. So Flux operator has these APIs that allow cluster admins to define how the ephemeral environment is created. It can be a namespace, it can be another cluster altogether maybe Cluster API or Crossplane. So you have like you have a wide array of choices of how you define ephemeral for your own environments and the Flux operator streamlines this dynamical behavior that is not in the nature of Flux. Flux is more static. It looks at what you have in Git. This is part of you know taking dynamic inputs from let's say the pull request API and then using the GitOps way it looks at what should I do now that I have this new information and maybe you would create a namespace, you would deploy a database, you would fill the database with some data that you need to test and then you deploy your app. So operator is not opinionated on how you do this workflow. It enables you to do this workflow and it's very much in line with how we designed Flux until now. Michael Bridgen: That's right. It sounds like it's sort of akin to how the image update stuff works, isn't it? You reflect the sort of external reality into the cluster and then Flux can react to that and do what you told it. Stefan Prodan: Yes. So in Flux we have the image automation feature which was since version one. And the problem with this approach is that you only react to code changes. You make a code change, you build a new container, you push the container to the registry. Then Flux image automation detects oh there is a new version of this container and let me push a change to Git. And it does a very narrow thing and we really did it because that's what Flux V1 had and so we did it for parity so people could still have that. So the problem with this approach is that if a code change requires a config change, there is no way to pair the two to deploy them in sync which is almost always what you want to do because especially if you add new features or you add a flag and so you need to provide a value for the flag otherwise the new image won't run. Exactly. Yeah. And Flux operator kind of solves this issue with image automation in a way that it does not write back to Git. It reacts to events that are happening around your Git repo like a merge request or pull requests are not Git things. These are abstractions built on top by the various Git providers out there. And we've implemented this so far but we are now looking at in the future with Flux operator. We want to allow this kind of dynamic provisioning to be controlled by third party software that can tell Flux operator, imagine you are building a platform, you're doing platform engineering, you're building your own platform inside your organization, you could integrate with Flux operator and drive these dynamic self-service environments from your own platform. And it kind of follows the Flux philosophy. We don't want to give you the whole thing. We want to give you the tool that enables and fits well with what you are building for yourself, right? It has the hooks that you can integrate with whatever, you know, however you prefer to do it, whatever tooling you're using. Michael Bridgen: Yeah, people have been wanting ephemeral deployments since Flux V1 like quite early on. Stefan Prodan: So it was one of the most requested features. We managed to deliver it in this Flux operator which is part of the Flux ecosystem. And it kind of builds on top of what we have today in Flux and I hope others will build their own integrations either through Flux operator or working directly with the Flux controllers. There are several platforms out there that build on top of Flux. For example Azure offers Flux as part of Azure Arc. It is built in, integrated into Azure UI, works with Azure DevOps and so on. So that's another example of how you can use Flux as your building blocks for the platform which is for end users. Michael Bridgen: That was always a goal with V2, not with V1 so much, but definitely with V2 was that, composable pieces that were also building blocks for making your own stuff, which you know that's starting to happen more so than it was. Stefan Prodan: So we have a roadmap that we try to publish for the whole year for the Flux project. And I can tell you a near-term goal, what's happening next in Flux. We want to double down on hardening what we already have, get our existing beta APIs to GA and especially we want to work more on the OCI artifacts integration. Then the whole movement where you know you can run Flux disconnected from a Git repo, don't need the Git server to be part of your production system. Flux can you know look at the container registry that's there near next to the cluster and that's where the desired state is being stored, signed and Flux can verify it. And so we've built a lot in that direction and I think what we are trying to do next is give users assurances if you, and there are many users that had switched to Flux with OCI artifacts including large enterprises that we are talking to right now. And we have this history where we've spent a long time iterating through the Flux APIs and beta versions and we always you know kept in mind backwards compatibility. We never wanted to break whatever people are doing with our beta APIs but we realized we can't keep them in beta forever. So the next goal is get the OCI artifacts and all the OCI integrations, all these features to GA. But it was also the case that when the OCI stuff was introduced, OCI itself was not that stable you know. It was kind of moot which direction it might go. So wasn't really possible before nowish. Michael Bridgen: Yes. Yes. I mean the OCI spec is a living breathing thing. I think the OCI, the implementations weren't really there, only supported Docker images more or less to start off with and then yeah I think to other things. Stefan Prodan: Yeah. OCI spec now has matured, all the implementations are really good here. We had the issue at the beginning, oh this container registry does not support OCI artifacts as a type. So we had the Flux CLI when you do a Flux push, it will push that not with type OCI artifact, will push it as a container image. We're just tricking the registry into accepting the payload that we want to have there. But now I think the container registry space has matured a lot. It's quite stable. We have support for everything we wanted in the Flux OCI integration. So I think it's the right moment now to go with this to GA and signal users if you really want to move away from Git and you want to adopt this new way of doing deployments which are let's say more secure, more resilient to single point of failures and so on, you can actually do it and it's fully supported. Another let's say major goal that we have is we put in place a new way of how you can become a Flux maintainer and we have this concept of a feature-focused maintainer where let's say you don't want to take on the burden of being a maintainer of everything in Flux, all the controllers, all the features, but you are let's say a person which are highly specialized in something or you really like a particular technology that Flux integrates with. And we have now this RFC process in place and we are building this in our governance where we can give you maintainership for the things that you feel like you are adept and you feel comfortable maintaining in Flux. And we already did that with Microsoft. We have a maintainer from the Azure Arc team and they are looking over all the Azure integrations, the GitHub integrations. And it proved that it's a good path forward because we have the goal of we don't want Control Plane to have all the maintainers. We don't want Microsoft to have all the maintainers, right? We want diversity in maintainership. And that's I think that's the goal for any CNCF project that has things about a long lifespan. It's the only way is to have a diverse group of maintainers. And yeah, one of the goals is try to involve more organization into this with this formula of don't be scared. You can come and take ownership only of the bits that you care about and help us drive the project for as a group of people working on various aspects of what's inside Flux. You know, if you have a feature in mind or you have some kind of architectural need, Slack, we have so many users there. There are many many discussions. So you'll put a question there, the discussion is interesting, lots of people respond, but the next day because there are so many, that information is kind of lost. So what we are trying to do is encourage people that hey for this type of discussion let's move it on on GitHub discussions. It stays there forever, it's searchable and it kind of builds a knowledge base outside of documentation through this with GitHub discussions, GitHub issues and stuff like that. So yeah, I think we have told our community like there are two ways, of course you can always get on Slack and ping us. But for you know more, when you want to tip over from just having a kind of chat about stuff into putting it on the record, and that point is if you want something you know to actually change about the code or yeah documentation. Also the RFC process where you say you want a new feature in Flux and how it all starts, you should open a GitHub discussion and find other users that find your idea appealing. And if you get that support then you will definitely convince any Flux maintainer to you know help you go through the RFC process and get the feature implemented, get it to GA and all of that. So that's a starting point basically. And people that have great ideas like you know you always want something new from a project but the fact is that we as core maintainers we have to balance sustainability, maintainer's burden. Are we going to implement 100 features this year? Are we going to be able to maintain them the next year? So yeah it's a more balanced approach that we are trying to do now with this feature-based maintainers and the new process that we have in place. Bart: From Michael's first commit to Stefan's V2 rewrite, Flux has come a long way. From a scrappy internal script to a production grade GitOps engine trusted worldwide. But code is only half the story. In our next episode, we hand the mic to the people running Flux in the wild across finance, telecommunications, and other high stakes industries. You'll hear how teams at Microsoft, at Orange are using Flux not just to ship software faster, but to stay compliant, secure, and competitive in some of the most demanding environments on Earth. That's next time as we explore Flux from an end-user perspective.