Comparing GitOps: Argo CD vs Flux CD

Bart: When it comes to debates around cloud native tooling, few things get more conversation started than the classic battle between Argo CD and Flux CD. In this episode of KubeFM, we dive deep into the famous rivalry between two of the most popular GitOps tools in the CNCF landscape, Argo CD and Flux CD. Our guest, Andrei, is a founder of Aenix and a cloud systems architect. In this episode, he breaks down the strengths and weaknesses of both tools, helping you understand which one might be the best fit for your team. We'll discuss user experience, access control, extensibility, and how each tool addresses different business themes. Plus, Andrei shares insights from his experience with Talos Linux, KubeVirt, and LINSTOR, as well as the importance of staying up to date with emerging Kubernetes technologies. This episode is brought to you by DigitalOcean Kubernetes. Now with NVIDIA H100 GPUs available on DigitalOcean Kubernetes, whether you're a startup building the next big AI model or an enterprise scaling complex machine learning workloads, DigitalOcean helps you scale from initial experiments to full production deployments with powerful, cost-effective infrastructure for your Kubernetes workloads. Explore more at digitalocean.com. Andrei, can you tell me which three emerging Kubernetes tools you're keeping an eye on?

Andrei: I can mention four projects. The first one is Talos Linux, a Kubernetes-specific distribution built with Kubernetes in mind. It runs as a Linux distribution on the node and is based on controllers, fully API-driven. You can configure the system by sending a configuration file, eliminating the need for SSH.

The second tool is KubeVirt, a Kubernetes add-on that allows running traditional virtual machines within Kubernetes. I appreciate how the Kubernetes API works and the patterns it implements. KubeVirt follows the same patterns, utilizing the Kubernetes API to run traditional virtual machines.

The third tool is LINSTOR, a storage solution that, although not very popular yet, is gaining popularity daily. I believe it is the fastest and most reliable storage in Kubernetes, capable of replicating data in the kernel. To use it, you need the DRBD kernel module. Once set up, you get fast and reliable storage for Kubernetes. It can also be used with KubeVirt, allowing virtual machines to consume LINSTOR-provisioned volumes.

The last project I'm keeping an eye on is etcd-operator. Although it's not yet part of the CNCF, the CNCF has started an initiative to create an official etcd-operator. Previously, we initiated a project to create an etcd operator, and now we're working to merge our codebase and donate the project to the CNCF. I need to keep an eye on this project to ensure a smooth merge and control the process.

Bart: Now to get started, can you just let me know what you do and who you work for?

Andrei: I'm currently working for myself. Previously, I worked for companies like Palark and Wedos, the largest Czech hosting company. I gained a lot of experience in operating bare metal servers, provisioning, and understanding the underlying infrastructure. When working with Kubernetes, especially in the cloud, you don't have to worry about many things like routing and volume provisioning. I'm an expert in networking and related areas, so I can advise on how to prepare a Kubernetes-based environment and run it on bare metal servers. I've started my own company, which packages this experience into a product. We offer an open-source tool that helps you get managed services out of Kubernetes provisioned on bare-metal nodes.

Bart: How did you get into Cloud Native?

Andrei: I contribute a lot to open source projects. I'm a member of multiple CNCF organizations, including Kubernetes, KubeVirt, LINSTOR, and Cilium. They recently applied our patch for force device detection, so I'm happy to communicate with these people. We're trying to organize our processes in a similar way to how it's done for CNCF projects. We have public meetings where everybody can join. I can say that we're an open source company and we're trying to build our processes to be open to everyone.

Bart: And what were you before Cloud Native?

Andrei: I was very open to the community. I was still writing a lot of articles. One of my articles helped me get to where I am now. I found the power of technical articles, as I can collect feedback and gain popularity. Many people saw my articles and videos, and I started visiting conferences. That was my activity.

Bart: And how do you keep updated with the Kubernetes ecosystem, the cloud native world, which moves very quickly. What works best for you?

Andrei: I visit a lot of CNCF meetings, for example, platform engineering meetings. We're also trying to focus on the projects we use to solve our own needs. In CozyStack, there are a lot of open source projects, and many of these projects belong to CNCF. It's always necessary to keep them updated. By attending these meetings, I watch for new technologies and sometimes try to solve specific needs. For instance, I've looked into how to access a service running in Kubernetes API, just by having access to the Kubernetes API. I've found existing tools like Mirrord or Telepresence that can solve this problem.

Bart: Just Google and you find the tool you need. If you could go back in time and share one career tip with your younger self, what would it be?

Andrei: Well, for now, I'm trying to build my own business. I would definitely say that you have to learn product view because it's essential to understand how to sell this technology and what really matters for business. Business provides the funding, and that's what drives technical progress.

Bart: Excellent. As part of our monthly content discovery, we found this article that you wrote, ArgoCD versus FluxCD. We can call this the great GitOps showdown. So we want to take this topic a little bit further with some questions. We've been recording the KubeFM podcast for a little over a year now, on our fourth season, and when we ask our guests about their favorite tools, Argo or Flux are likely to be part of the answer. Why do you think that's the case?

Andrei: I think both tools are good for their own purpose. So, yes, if we're talking about GitOps tools, they're both GitOps tools, but they do their job in different ways. To answer this question, I can explain what GitOps is and what process it solves. When we talk about GitOps, we usually talk about continuous delivery. If we need to deliver software to an environment, we can split this process into CI and CD - continuous integration and continuous delivery. In CI, we have a set of actions needed to get a predictable artifact, and CD is the tool used for delivering this artifact to the end environment. ArgoCD and FluxCD both do CD, so if you have an artifact, such as a Helm chart or Kustomize, they know how to deliver it to the end environment. However, they do it slightly differently.

ArgoCD is good for developers and is cheap for administrators to implement in their infrastructure. If you need to organize development and help your developers deliver their application to production, staging, or dev environment, ArgoCD might be really good because it provides many features that can be used out of the box. On the other hand, FluxCD is very good for administrators. For example, if you're delivering infrastructure services like cert-manager, GitHub, NGINX ingress controller, Prometheus, and other infrastructure components, FluxCD works better. FluxCD can also be used for preparing reproducible environments, which is why we use FluxCD for preparing our platform. You can use FluxCD to bootstrap 100 clusters and fill them with the same configuration. However, FluxCD is more likely to be automated, whereas ArgoCD is more about interacting with developers.

Bart: With that in mind, how do Argo CD and Flux CD implement GitOps?

Andrei: They do it differently. We can say that ArgoCD is usually used in a different way. ArgoCD implements a pattern where you have one single ArgoCD instance, and you can add multiple clusters to this ArgoCD, using it as a central point for controlling all of your deployments. On the other hand, FluxCD can be used as a component of your cluster, implementing the Kubernetes controller pattern in Kubernetes. It extends your Kubernetes to add the opportunity of storing the desired state in Git or other sources. For that reason, ArgoCD has a really good, beautiful UI, which you can give to your developers to see the process of your deployment. FluxCD, since it is based on Kubernetes, is more like a Kubernetes add-on. It has a really cool CLI interface, which allows you to quickly connect FluxCD to a Git repository. It can automatically issue tokens and configure your cluster to use a Git repository.

Bart: Now, you wrote an article that compared Argo CD and FluxCD. How did you devise a list of comparison criteria, and what are they?

Andrei: First of all, if you're considering these tools, you need to understand the business need you're trying to solve. If you want to provide a simple interface for your developers, deliver code, and manage the release cycle, or have staging, production, and development environments and just push your code into them, ArgoCD might be better because it provides everything you need to quickly start doing this. However, if you want to configure reproducible environments, for example, if you have multiple Kubernetes clusters and you want them to be similarly configured with similar components, FluxCD might be better. I think FluxCD is very well-suited for delivering system components and Helm charts. It's also nice if you want to build reproducible environments or platforms with components that you're going to use in multiple Kubernetes clusters.

Bart: Let's take a closer look at the interface and user experience. Wouldn't we say that this is an easy win for Argo CD?

Andrei: From one point of view, yes. When considering how people choose technologies, they first look at the interface and how it looks. ArgoCD looks very well and is easy to start using. You can install it in a Kubernetes cluster, access the web interface, and configure everything you need. ArgoCD provides many features out of the box, allowing developers to control and manage their applications. They can automatically add Git repositories, view logs, restart pods, and even execute commands in a pod, which can be really useful. One of the killer features of ArgoCD is that it eliminates the need to provide access to the Kubernetes API for developers. Instead, they can view the state of the deployment directly in the ArgoCD dashboard and take necessary actions.

From another point of view, this approach is not very declarative. FluxCD, on the other hand, forces you to keep all configuration in Git and describe it in Kubernetes resources. You can use FluxCD for developers to deliver business applications, but it may require additional setup. For example, to provide logs, you need to organize access to Grafana or Elastic. There are several issues like this, especially since FluxCD is a framework that does not provide an end-user interface. However, it uses native Kubernetes resources, which is a cool feature. Additionally, it has a really cool CLI interface that allows you to connect a Kubernetes cluster to a Git repository. This can be really useful for administrators who work with the command line and are familiar with Helm charts. Connecting a Git repository storing Flux configuration with Kubernetes should not be a problem for them.

Bart: To follow up, does a graphical user interface make onboarding easier in a Kubernetes cluster? Could we say that Argo CD is easier to start with?

Andrei: I think yes, but you can face difficulties. For example, ArgoCD's approach is not very compatible with existing Helm charts. ArgoCD has its own RBAC model, so you can configure roles the way you think would be better for ArgoCD, but it's not the same as the roles in Kubernetes. They implement a different approach for this. It might be difficult to install ArgoCD and configure it declaratively, because the configuration format is like CSV files inside YAML, which looks really weird. I think ArgoCD is intended to provide a web interface and configuration through the web interface. One time, we had a flow where we had two ArgoCDs, one for staging and one for production. We configured the staging ArgoCD through the web interface, then downloaded the configuration from staging and put it into production with some modifications. However, if you want to just try, I think ArgoCD is a good thing to start with. There's still a question about what problem you're trying to solve. If you want to configure your cluster to install common charts like cert-manager, Prometheus, and others, it might be better to use FluxCD. With FluxCD, you can simply connect your cluster to a Git repository and specify which charts with which parameters should be deployed, and it will deploy all these Helm charts automatically. This is very cool. So, I think ArgoCD is good if you're looking for a tool to help deliver your own application. But if you're a Kubernetes administrator and want to deliver a standard application stack into Kubernetes, choose FluxCD.

Bart: Is FluxCD better if you have a lot of Helm charts?

Andrei: This is a slightly weird question. I think I have both. ArgoCD can also work with a lot of Helm charts, but they do it in a different way. If you want to deploy official, standard Helm charts in ArgoCD, they usually need to be fixed somehow. FluxCD uses native Helm to install Helm charts and native tools, so you can see them in Helm LS, for example. You can operate them the same way as you would using Helm, but in a declarative format. From my point of view, FluxCD should be better at scale. That means if you use ArgoCD, it is intended to have a single ArgoCD instance for everything. In contrast, with FluxCD, every Kubernetes cluster should have its own FluxCD controller installed.

Bart: Most companies might not want to grant unlimited access to deploy anything they want. How do Argo CD and Flux CD handle access control, and what options are available to restrict deployments?

Andrei: This is an interesting question because one of the powers of GitOps is that it allows you to inspect what exactly is happening right now. If you make some changes or deploy something in ArgoCD, you can always see the diff between the desired state and the existing state in the cluster. It will show you exactly what will be changed, such as which lines in the Chrome manifest will be removed and which ones will be added. You can control this in the deployment phase, but GitOps also means you can do this at the review phase, before the changes are merged into the repository branch from which the GitOps tool is syncing. In this way, you can prepare a model where, for example, if you have a Git repository and your GitOps operator is looking for changes in the main branch, you can configure the GitOps operator to watch for changes in the main branch and restrict any pushes to this branch, following a pull request model or merge request model. This way, you can control all the changes that get into the main branch and predict exactly what will happen.

Bart: Is Argo CD's idea to reinvent RBAC good?

Andrei: I don't think this is a big trouble. Having CSV files in YAML is not convenient, but they still need to store this data somewhere. It's cool that they're storing this in a configuration file. I think it might be worse if they're storing configuration in a database. While it might be difficult to template, it is still possible. You can configure ArgoCD in a declarative way. You might feel the pain at the beginning, but if you come up with a good interface for this, you can just template it and use it as is. On the other hand, we can see that FluxCD fully utilizes the Kubernetes controller pattern. It has its own custom resources, and all the configuration is stored in them. Even the standard Kubernetes Custom Resources model works for FluxCD resources.

Bart: Now, let's talk about extensibility. Engineers might want to integrate ArgoCD or FluxCD to pull secrets from their secret stores of choice. Or they might want to customize the tool used to template Kubernetes resources. How can they extend ArgoCD and FluxCD to support those use cases? This is a good question. I think ArgoCD, for now, does not provide any certificate management, so...

Andrei: You have a few options to customize ArgoCD. You can provide additional configuration plugins or use secrets that expand themselves in Kubernetes. For example, you can use Sealed Secrets, putting a common encryption key in Kubernetes that will be unsealed every time you apply these secrets through ArgoCD. This is one solution. From another point of view, FluxCD has an integration with SOPS (Secrets OPerationS), a nice and easy tool for keeping secrets in a Git repository. It allows you to use it as is, which is very cool. I really appreciate that the FluxCD maintainers implemented this feature out of the box.

Bart: And in your particular case with CozyStack, why did you choose FluxCD for your platform?

Andrei: Mainly because of the standardization, we really like how it works, and we believe this is a feature for our platform building. However, we don't use FluxCD in the usual way. We implemented our own approach, where FluxCD is used just as a Helm operator. We deliver a container with a Helm repository and all the needed Helm charts that need to be installed in Kubernetes. Then, FluxCD syncs the information from those Helm charts and installs them into the same cluster. This allows us to create reproducible environments and do releases that update all the components in the cluster. We also implemented a very cool development experience for our platform, so you can use standard dependencies. For every chart, you can do local development and see what will be changed after you apply it. However, we still use ArgoCD to organize the delivery of business applications for our customers. Even though we use FluxCD for delivering infrastructure components of the platform, it is also used for the dashboard and for the applications they choose to install. But if they need to organize a development process where a developer would see the process of deploying a business application, they use ArgoCD because they really like how it works and how it looks.

Bart: And now the million-dollar question, Andrei. We've spoken about different aspects of getting the two tools, but I promise I won't tell anyone if you have to choose one - which one is the best?

Andrei: I can't say for sure, but I think it depends on your business needs. Both tools are very cool and good from my point of view. You have to use them for specific purposes. ArgoCD is good for developers and FluxCD is good for infrastructure administrators.

Bart: You are a busy person, but you still have enough motivation to write these articles. What was the primary motivation behind that? Were you fed up with all the Argo CD versus Flux CD fights that are common on Reddit?

Andrei: I recently came across a dispute in the community about which tool is better. I thought, "Hey, both tools are cool." So, I decided to write down some points. As I wrote, I realized there were so many points that I decided to put them into an article and publish it, as I always do. I like to write technical articles to summarize my experience. This approach also allows me to collect feedback from the community when I'm unsure about something, which can be really useful. Sometimes, I even do some research, like when I write an article about a technology I don't fully understand. This process allows me to find the right people, and they can correct me. I really appreciate how the community works in this way.

Bart: And on the article that you wrote, did you get any feedback? Did any fights start? How did the community react to your article?

Andrei: Especially this article was very popular, I think, because of the versus. People really like to read articles that compare technologies. I have experience writing about and comparing Kubernetes storage options. This article was also very popular. So, if you want to write a popular article, just compare some technologies.

Bart: What's next for you?

Andrei: I'm working on my business and I want to put it on track. We're very successful at this phase, and for now, we're looking for investment. We have a growing number of customers and a really cool team. The main point for me right now is to finish setting our business on track. Of course, I want to continue contributing to open-source projects. The technologies we use in CozyStack are all open-source, and we have partner relationships with them. We have cross-marketing activities and promote each other. This is very cool because many of the technologies we use are not very popular and still need to gain more popularity. CozyStack combines all these amazing technologies together, and now we're conquering the world this way.

Bart: What's the best way for people to get in touch with you?

Andrei: You can follow me on LinkedIn, Twitter, or Medium. Check out our product at CozyStack. The best way to get involved is to attend our community meetings, which we hold every week. These meetings are open to everyone and cover really interesting subjects. You can also influence the product itself by joining the discussion.

Bart: Andrei, thank you very much for joining us today. We're looking forward to the next steps in your company and the articles you'll be writing. As always, being a community-focused person, you're giving back so that others can learn and make the right choice depending on their use case. Keep up the great work.

Andrei: Thank you. Bye.

Bart: Bye. Take care.