Bart Farrell: At small scale, a deployment pipeline that runs Helm upgrade directly against a cluster can feel simple and effective. But as the number of teams, services, and environments grows, that model starts to expose serious problems. Configuration becomes buried inside CI logic, manual changes create invisible drift, and the pipeline, not Git, becomes the closest thing you have to a source of truth. At WSC Sports, Elad Cohen's team supports roughly 200 developers, hundreds of microservices, and around 60 Kubernetes clusters. In this episode of KubeFM, Elad explains how they redesigned their deployment platform by separating CI from delivery, moving from Azure DevOps to GitHub Actions, and handing reconciliation over to Argo CD. We get into reusable Helm charts, ApplicationSets, self-healing, shadow deployments for integration testing, code-freeze controls, emergency deployment paths, AppProjects for team-level permissions, and how a single values file can create a complete deployment workflow. This is a practical look at what it takes to make GitOps work at enterprise scale, and why the hardest part is often not changing the tools, but changing the operating model around them. Kubernetes requests and limits are easy to guess and hard to get right. Set them too high, and you waste capacity. Set them too low, and you risk throttling, OOM kills, or downtime. LearnKube has put together a free set of resources to help: a book, interactive calculators, and a cheat sheet for getting CPU and memory values right. You'll learn how to identify over-allocated containers and decide what is worth changing safely. Download them at learnkube.com/kubernetes-rightsizing. Now, let's get into the episode with Elad. Welcome to KubeFM. First things first, what three emerging Kubernetes tools are you keeping an eye on?
Elad Cohen: The first tool is one I use myself. It's called Aptakube, which is a Kubernetes GUI. It's a paid subscription one, so keep that in mind. I prefer it over Lens because it's much faster and it allows better multi-cluster context. So that's a great tool to check out. The second tool is Argo CD Agent, which is a new Argo CD in development. It's not yet production ready, but it has one control plane on a master cluster with Argo CD. And it has many runners on each of the hosted clusters, which means that if I have 100 clusters, this solution will work much better because the compute will happen on the clusters themselves. But since it's still in development, it's not yet available for use. So just keep that in mind. And the third one is Kite, which is a web UI for Kubernetes management. The thing with this is that when you have a lot of devs, like 100, 200, 300, and more developers, you don't necessarily always want them to install an IDE on their computer. So Kite is a Helm chart that you install on the clusters, and it allows you to do all the IDE stuff that you need inside the web. It works with Entra ID for authentication, and it's really nice.
Bart Farrell: For folks out there who don't know you, can you tell us a little bit more about who you are, what you do, and where you work?
Elad Cohen: My name is Elad. I do DevOps and platform engineering at WSC Sports. A bit about the company: WSC Sports is an AI-powered platform that generates sports highlights in real time. That means, for example, if there are baskets in the NBA by LeBron James, and those baskets were really nice, it automatically ends up on our platform. The NBA editors can just create it and post it to social media straight from the platform without editing or cutting. It all happens automatically using AI. A bit about my role: I cover the entire infrastructure layer, which means I'm handling all of the DevOps responsibilities in the organization. I manage Kubernetes, GitHub, Argo, all the monitoring, all the cost monitoring, DevEx, and all those worlds as well. I have around seven years of DevOps experience. That's pretty much it.
Bart Farrell: It's nice to have that context about where you work. It's something that we can all relate to. We want to see the best highlights of LeBron James scoring baskets in real time. In terms of your background, Elad, how did you get into Cloud Native?
Elad Cohen: In my first role, I was managing a private cloud, which means they had VMware solutions, NetApp solutions. I had tens of thousands of VMs. The scale just got so crazy that we had to figure out new solutions. We understood that it took way too much work. It was barely automated. That's how we understood that transitioning into microservices could be the move. Once we made the decision, it was my training ground. I built clusters from zero. I used Rancher. I used RKE. That's where I first started using Argo CD. That's how it happened.
Bart Farrell: What were you before cloud native?
Elad Cohen: I was managing large NetApp systems that included NFS file shares, SMB shares. I worked with virtualization, data center hardware. For example, I was managing plenty of Cisco servers and I installed the virtualization layer on them. I was doing CI/CD with Jenkins and GitLab, which now looks like something so old that it's crazy how time passes.
Bart Farrell: Now, the Kubernetes ecosystem moves very quickly. In your case, how do you stay updated with all the different changes that are happening?
Elad Cohen: My favorite is the TLDR newsletter. Every day it sends new articles, new upgrades, new versions of tools that we use day to day, which is very nice. Because I'm registered with Medium, I get some articles like the one I wrote, and other stuff that's related to me. I'm also in the CNCF Slack, where you can advise someone or hear about new release candidates. They always post it there.
Bart Farrell: If you could go back in time and share one career tip with your younger self, what would it be?
Elad Cohen: I'm very excited when I see new tools. I always want to get started and test the tool and see how it fits. I think it's a better practice to understand the problem first and then find a tool that solves it, rather than looking at a cool tool and implementing it straight away. The second thing is focusing on projects involving things I don't know. It becomes easy after you know some subjects very well: all of your projects will focus on them, and you'll perform the best you can, and all of the artifacts that you produce are great. But for personal development, always look to do what you haven't touched before.
Bart Farrell: As part of our monthly content discovery, we found an article that you wrote titled GitOps at Enterprise Scale: Inside WSC Sports Deployment Platform. We want to dig into this topic further. But before we get into the deployment changes, let's establish the baseline. What does the engineering and Kubernetes environment at WSC Sports look like in terms of teams, services, clusters, and the general path from code to production?
Elad Cohen: WSC today has around 220 employees in the R&D department. There are between 15 and 25 development teams. We run a large microservices architecture. Most of our workloads on Kubernetes are event-driven, which means we have a lot of queues, storage queues, and service buses on Azure, from which most of the workloads take jobs. We're mostly on Azure. We have 60 Kubernetes clusters, both development environments and production ones. In terms of ownership, every team has their own services, and they own them end-to-end, which means they're responsible for deploying them, creating the pipelines, operating them, checking logs if they have any issues, and creating all the infrastructure in our Terraform repos. For the path from code to production, we have GitHub Actions for CI; we have Argo CD for the delivery to the cluster; we auto-sync everything that we can, except for automated prune for safety, which means a name change can't delete the old Deployment. At our scale of hundreds of microservices, having an automated deployment model is survival. After all, we're around 10 or 12 DevOps engineers for 200 developers. We need to have everything as automated as possible.
Bart Farrell: Once we understand the environment, let's zoom into the original deployment workflow. Many platform teams tried to reduce friction by giving developers a standard path to production instead of asking every team to build its own process. How was deployment structured at WSC Sports before the redesign?
Elad Cohen: All of our pipelines were in Azure DevOps, as were all of our repos. We had one template library, we had a repo called DevOps Pipeline Templates, we had a Kubernetes template, we had an app service template and we had some other ones. When you created a pipeline, you had to provide a small set of inputs such as image name that you want, the environment you're deploying to, requests and limits, and some other configs. The pipeline did everything: the Docker build, the Docker push, all the environment configurations, and Helm upgrade into the cluster, which means from Azure DevOps, we had multiple service connections for each subscription in the Azure, which connected to the clusters and did a Helm upgrade every time. From a developer perspective, it was pretty clean. You didn't need to understand any internals or the infrastructure layer. From the platform perspective, it was centralized and consistent, which means it worked well. We could scale easily. Every team could handle their own services and create their own pipelines, which was a win at the time.
Bart Farrell: A centralized workflow can be very successful at first because it hides complexity and creates consistency. But the same abstraction can also hide architectural problems until the platform grows. When did you start seeing limits in the existing model?
Elad Cohen: As the platform grew, we had more teams, more services, more edge cases. We needed to add many parameters. For example, one team needed ephemeral volumes for their Pods, which wasn't supported. Each time, we had to keep adding them into the centralized template, and it was much harder to maintain: with hundreds of pipelines, you're not sure what's defined where, and it's getting hard to track. That was the first issue. The second issue was that we didn't have any source of truth. A pipeline that runs Helm upgrade, for example, just changes the cluster. After that, the resulting state exists only in the cluster, and you have no visibility into it. When the pipeline is setting state once, but not reconciling it, you have a problem at large scale because you can't control all the environments.
Bart Farrell: Now, Kubernetes gives teams a way to describe how the system should look, but many pipelines still work by running the command at deploy time. In your previous setup, how did you think about the source of truth for what was actually running?
Elad Cohen: As I said, the source of truth was the pipeline. All the cluster showed was whether the last Helm upgrade worked and when it happened. Other than that, you didn't have any idea if someone changed the scale. It could be environment variables. It could be node tolerations. It could be many things that were there until the next redeploy. There was nothing watching and correcting the cluster over time. For example, one edge case that happened to us: sometimes two deployments ran at the same time, or one or two minutes apart, and one build took a bit longer; the developer could come to us and say, "My change wasn't deployed, I don't know what happened." Only once you go to the pipelines and see that the deploy actually started a minute earlier, you understand that you don't know anything. That's when it hits you. That's what makes you realize something needs to change here.
Bart Farrell: Drift is one of those problems that often stays invisible until there is an incident or confusing production state. What kinds of changes could happen outside the normal deployment path, and how did that affect trust in the system?
Elad Cohen: Manual kubectl changes, including scaling, editing a ConfigMap, or patching something during an incident, are invisible to the pipeline. With Terraform, on your next apply or plan, you see what's changed, what's the actual state and what's the desired state. Here, you're doing a Helm upgrade and you pray nothing's changed during an incident. On the next deploy, all the changes are overwritten.
Bart Farrell: Configuration often starts as something simple inside a pipeline, but then over time, teams add environment rules, runtime settings, exceptions, and deployment behavior. What made configuration management difficult in your previous setup?
Elad Cohen: What makes it difficult is that the configuration of the infrastructure lives inside the CI YAML. It's mixed in the pipeline logic. For example, I want to change a resource limit. I'm changing the configuration in the CI YAML and in order to trigger this change, I have to rebuild it and redeploy it to the correct environment to change something in the infrastructure that has no relation to the code. Config changes to the infrastructure have to be completely separate. There's no reason to rebuild the whole application to change a resource limit. It worked, but it was getting harder to reason about this behavior as we kept scaling.
Bart Farrell: Now, the move from Azure DevOps to GitHub Actions could have been just a translation exercise. What made that project turn into a broader rethink of the deployment platform?
Elad Cohen: We started with the intention of translating our pipelines using the same logic but with GitHub Actions syntax. There are tools that do exactly that. When you rewrite something and copy all the commands, you see each behavior and ask: "Should we keep it?" You may not get to it later. Once it stands and the services start to migrate, changes become difficult. We kept asking, "Why does it work this way?" Many answers were because that's how we did it and not because that's the best way to do it. That's something I wanted to address. The migration is an opportunity. You pay the cost of rebuilding. You might as well build the right thing.
Bart Farrell: When you redesigned the pipeline, how did you separate the different responsibilities in the release process?
Elad Cohen: CI and CD are completely different things. They should be separated completely. The build workflow checks out the repo, it does the build, it pushes it and outputs the image tag for the deploy. It has no idea about environments and things like that. It's generic across all deployment stages. For example, if I have a DR region, we use the same build artifact and the same image tag, but with a different env var that allows me to use configuration for the specific region. That's why we kept the build as is; we didn't have to touch it. The deploy workflow is a completely different reusable template. It has pre-deploy checks, it has approval gates for the team before deploy. It has testing and the deploy step itself, which is the CD, is patching a new image tag in a Git values file. The separation is as prominent as it gets. That makes the responsibility clear between the CI and the CD.
Bart Farrell: With that in mind, production delivery usually has extra constraints such as freezes, approvals, emergency fixes and different rules from development. How did you build those controls into the new process without making deployment feel completely manual?
Elad Cohen: We have code freezes on some days of the week. It's usually when we have large sports events, when we need the production to be extra stable. The first gate that I had is the code freeze check. In the approval gate step, whenever we have a code freeze day, it's blocked. This is the first protection that we have. If something is urgent, we have a workflow dispatch that you can run manually and you can set a flag that bypasses the freeze. It requires an extra approval by a team lead. You need one approval by the team lead to even start this workflow, and then you have to approve it again before the deploy, like a regular pipeline. Once you do that, you get a Slack message in a general channel that says something was pushed during code freeze in case any bugs occur because of that. For hotfixes, we care mostly about velocity. We flip the test order. We run pre-deploy tests post-deploy. Then we verify everything is good. In general, dev deployments skip all of that. If I'm building something in my dev environment, I don't have to approve anything. All of the deploys happen automatically. There's no friction for that.
Bart Farrell: Now, once CI stopped directly changing the cluster, something else had to take responsibility for moving the system toward the intended state. How did you design that handoff?
Elad Cohen: One conceptual change was that CI no longer touches the cluster directly. It only touches Git. Previously, in Azure DevOps we went into the cluster and did a Helm upgrade command; now CI doesn't. All of the clusters are connected to Argo CD and all CI needs to do is patch a values YAML file and that's how the tag changes. Then Argo CD picks that up using a webhook. We also have one limitation, though I don't remember the exact details: only the changed app is refreshed. We added it to the ApplicationSet. Once we have the webhook, it re-renders the Helm chart, it applies the rolling update to the cluster, and that's how the update happens. The key addition over the old setup is self-heal set to true, which means the reconciliation is not waiting; if you change something, it will be changed back immediately. That's how Git becomes the real source of truth. The cluster's job is to match it continuously.
Bart Farrell: Now, testing before production is always tricky when the application needs to run inside Kubernetes. How did you give teams a realistic test path without touching the live service first?
Elad Cohen: In Azure DevOps, we had a shadow deployment, which did a Helm upgrade to the shadow deployment on one of the Dev clusters. Then integration tests were run, and once they were successful, there was another Helm upgrade that scaled it to zero, followed by a new Helm upgrade to deploy the actual service. I wanted to mimic this behavior in the new setup, so I added to each values YAML an automation block with a specific tag and a specific scale. Once the CI runs, the tests start. The tag and the scale in the automation block of the specific values YAML are patched. It waits around a minute until everything is up. It does the tests. It patches it back to scale zero. Then it moves into the actual deploy. This provides clean isolation. We get a realistic test environment with this new Pod because most of the things are event-driven. We have different queues for automation jobs. That's how we know jobs are working well.
Bart Farrell: Any test system has failure modes. CI can fail halfway through, resources can be left behind, or someone can manually interfere. How did you make the testing flow resilient to those kinds of problems?
Elad Cohen: The shadow deployment is now part of the declared desired state in Git. Before that it was a bigger issue: if someone lowered the scale, changed the image or did things like that, but now it's being reconciled. For CI failures, whether it fails the tests, it is canceled, or encounters any general failure, the scale-down job always runs. If someone manually changes the automation Pods, the reconciliation solves it. That's how we don't have to do any manual intervention. Nothing is left over and no orphaned resources remain.
Bart Farrell: Now a platform like this needs a standard contract between service teams and the platform team. How did you go about deciding what every service should define and what should remain optional?
Elad Cohen: We built a single Helm chart called core_deploy. All the services use that. The required fields are minimal: the chart name, service name, namespace, requests and limits. These are mandatory, but some services use things that others don't. Examples include GPUs, Secrets Manager integration, and node scheduling preferences; those aren't mandatory. Our principle is we don't require teams to think about things they don't need yet. Once you need something, it arises, and then the solution is already waiting. If a team needs a feature, they add it to their values file. If they don't, they never see it. Everything else uses defaults.
Bart Farrell: Onboarding is often where platform abstractions either succeed or fail. What does a new team or service have to do to start using this deployment platform?
Elad Cohen: This is one of the cool features. All they have to do is create one file in a repo. Under applications, they set environment, team, and the service. Inside that, you can have either one values file for a specific region, or you can have three. You can have values-EastUS, values-EastUS2, values-EastUS3. All of those will work. An ApplicationSet defines the pattern of values-*.yaml, and it automatically creates them. They also specify a field called chart. They can have either core_deploy or they can have core_deploy_cronjob or other charts that they need, although those are used most often. If they need integration tests, they add the automation block inside the values in the same file. Once they merge it, two Applications in Argo CD appear automatically. The automation ApplicationSet and the regular core_deploy ApplicationSet create them. Permissions and Slack routing are already in place because of the AppProject of the team. After that, they have their deployment set up. They then create the CI workflow from the template. It has two inputs: the build parameters, such as the solution's location in the repo and the image name, and the deploy part, which is the path to the values file. That's it.
Bart Farrell: There is also an automation layer behind the scenes that turns service definitions into running applications. Without assuming everyone has built this kind of setup before, could you walk us through how the platform discovers services and creates the right deployment objects?
Elad Cohen: We have two ApplicationSets in the Argo CD repo. The first one scans all the values YAML. It pulls values directly from the matched file using Go templates. If the ApplicationSet needs a specific definition for the Application itself, it passes it through to it. The article contains the exact example. The second one does the same for the automation shadow deployment. It recognizes that it has an automation block, and it uses the same charts and namespace, but with KEDA disabled and the scale set to zero by default. Both ApplicationSets use self-heal, so all the Applications are reconciled all the time. The result is simple: you merge one file, two managed Applications in Argo CD are created. This produces two Deployments and the service is up in production.
Bart Farrell: Multi-team platforms also need clear boundaries or guardrails, as some would call them. How do you give teams enough access to operate their own services without giving everyone broad access to everything?
Elad Cohen: We're using a feature called Argo CD AppProjects. In discussions with colleagues, I found a lot of them don't use this feature, so they're missing out on a great feature. It allows my Argo CD to split into spaces. For example, for a DevOps team, I set the Argo CD project in the values YAML to the DevOps project. There I can add some limitations. For example, I can set that DevOps projects can only build applications on DevOps clusters or DevOps namespaces. This is useful when one cluster is shared by many teams. This allows us to set permissions according to the project. For a DevOps team, I can set exactly which permissions they need. For example, I want them to only read, sync, and view logs, but I don't want them to delete anything. I configure it there. It also connects to the SSO groups that I set in Entra ID. We create a project for each team, routing notifications to this team's Slack channel, and setting permissions in Argo CD. An admin can filter the project and see all of those application boxes. The development team only sees their applications. This creates a great permission model. For notifications about degradations or failures, sync failures, unknown states, each project can use a different Slack channel.
Bart Farrell: Platform standardization can sometimes create tension with teams that need customization. How did you handle requests that did not fit the existing service contract?
Elad Cohen: When a team needs something new in their infrastructure, we need to think as engineers whether it's something that will pop up in the future for other teams or it's just that team. In my experience, teams hit similar needs at different times. Something one team requested a year ago could be the same thing another team requested from me a week ago. My approach is to extend the reusable templates, such as the core_deploy chart. That means the fix will also be available for the team that requested it, but in the future, if another team needs the feature, it will be easy to set up. Over time, that's a compounding benefit. The benefit becomes visible: the charts support GPUs, KEDA, Gateway API, ephemeral storage. Other capabilities that teams may need in the future are already set up.
Bart Farrell: Now, Elad, looking back, this seems less like a tooling migration and more like a change in operating model. What would you tell another platform team that is considering a similar move?
Elad Cohen: The hard shift isn't technical; it's conceptual. Changing to a mindset where Git is the source of truth is hard for a lot of devs. In my experience, I give them sessions every month or two about these concepts. This was one of the things that they did not accept fully until they tried it and saw the benefits. That's the first lesson: you need to be prepared for a move like this. Also, don't copy old pipelines into new tools. Treat every big migration that involves a lot of people and a lot of changes in the infrastructure as an opportunity to rethink deployments from first principles. Extend centrally. Every time you get a request for a change, make it easier for the next time that someone requests the same thing. Put it in a template and automate it. One rule is: if something takes over 30 minutes, automate it.
Bart Farrell: Very sound advice. Now, what's next for you?
Elad Cohen: My next few projects are more classic infrastructure work: building new architectures, using front doors and caching mechanisms, and stepping a bit out of the DevEx zone. As I said at the beginning, do things you're less strong at, not what you're the best at.
Bart Farrell: If people want to get in touch with you, what's the best way to do that?
Elad Cohen: I'm available on LinkedIn. Just DM me.
Bart Farrell: Perfect. Thank you so much for sharing your knowledge with us today through your work in the article and the depth of the answers that you provided. I hope our paths cross again in the future. In the meantime, best of luck with your new projects. Take care.
Elad Cohen: Thank you very much, Bart.
