Running Kubernetes On-Prem with AWS

Running Kubernetes On-Prem with AWS

Mar 31, 2026

Guest:

  • Raglin Anthony

Running Kubernetes on-premises sounds straightforward until you're managing control plane upgrades, certificate rotations, and multi-cluster networking on your own. The operational burden adds up fast.

Raglin Anthony, Solutions Architect at AWS, shares how organizations can bridge the gap between on-prem Kubernetes and cloud-native practices — without a full migration.

In this interview:

  • Options for regulated environments: from self-managed clusters with KubeADM to EKS Anywhere for air-gapped deployments

  • The hidden operational challenges of on-prem Kubernetes: control plane management, DNS, certificate management, identity fragility, and observability gaps

  • How on-prem to cloud networking works: public internet, AWS Site-to-Site VPN, and AWS Direct Connect — and why DNS always comes last

  • EKS Hybrid Nodes explained: how it removes control plane burden while keeping worker nodes on-premises, with native IAM and EKS Pod Identity support

The common thread: the operational complexity of Kubernetes doesn't disappear on-prem — but the right tooling can shift where you spend that effort.

Subscribe to KubeFM Weekly

Get the latest Kubernetes videos delivered to your inbox every week.

or subscribe via

Transcription

Bart Farrell: So who are you? What's your role and where do you work?

Raglin Anthony: My name is Raglin Anthony. I go by Ragz. I'm a special solutions architect at AWS. I work in the media and entertainment, gaming and sports industry. And I'm based in London.

Bart Farrell: Very good. You're based in London. I'm based in Spain. It's not too far away. What part of London are you in?

Raglin Anthony: Well, I work in the London office in Holborn.

Bart Farrell: Okay.

Raglin Anthony: And homies in Reading.

Bart Farrell: Okay, cool. Good. Now, what are three emerging Kubernetes tools that you're keeping an eye on?

Raglin Anthony: So emerging tech, I guess The obvious one is going to be KRO, Kube Resource Orchestrator. There's a lot of changes happening in that space. We recently had EKS capabilities supporting that. So I'm really interested to see how that project grows, how the community adds to that in terms of bringing together further automation within Kubernetes clusters. The second that I'm looking at is Kueue, more specifically Multi-Kueue for job scheduling. So I spend a lot of time working through AI, ML workloads and in terms of training jobs. And with the consumption in terms of GPU usage, Multi-Kueue seems like an interesting package to be able to utilize clusters across different regions. And the last one that I'm interested in is KubeVirt. So KubeVirt is really interesting to me. Specifically, we've recently had EC2 support nested virtualization, so I'm keen to see how we can use KubeVirt to manage virtual machines within EC2.

Bart Farrell: What are the options for organizations with regulatory requirements that prevent immediate cloud adoption but want modern Kubernetes practices?

Raglin Anthony: So for organizations that work in regulated environments where data residency, sovereignty, and auditability is almost like a non-negotiable, there's a couple of options, I would say, for running Kubernetes. The first one would be a fully DIY, self-managed Kubernetes cluster using tools like KubeADM. That'll give you full control of the entire stack control plane and data plane. But that also comes with the operational process for managing those clusters. You can also use vendor managed solutions. And that brings some tooling to take away some of the operational processes with a DIY type of cluster. But then you're working within the boundaries of what that particular vendor supports. From an AWS perspective, we've got a couple options as well. So you could start with AWS Outposts where it's AWS managed infrastructure in your data center. And then you get a very cloud-like experience on-premises. But then that comes also at a higher cost and a longer lead time to get the cluster up and running. Another option is EKS Hybrid Nodes. So Hybrid Nodes is your infrastructure in your data sensor being used as worker nodes, and where we've got the console plane running in AWS via EKS. And that's a great option to leverage AWS from a control plane perspective and still give you control over the worker nodes on-premises. And the last deployment option I would say is EKS Anywhere. And EKS Anywhere is going to give you AWS tooling and AWS APIs on-premises. It's suitable for air-gapped environments. So that means you're going to be running the data plane and the control plane on-premises.

Bart Farrell: For teams already running Kubernetes on-prem, what are some operational challenges they face on a day-to-day basis?

Raglin Anthony: Okay. So control plane management, I guess, is the big one. So patching, upgrades, scaling, securing the control plane, that's all on your team. The other, I would say, is networking. You know, especially when you've got multi-clusters, being able to manage your IP ranges, being able to handle your DNS across cluster services, that can become complex, especially when you also need to call out to like cloud services. Another sort of low level anxiety could be like certificate management. It's very easy for that to stall cluster operations. Identity and access management can become a bit fragile because the teams don't have a process in order to handle identity and access management. It's very easy to fall onto like static cube config files, maybe using a managed service account of some sort, possibly even getting to the point where you've got to bolt it on OIDC. And then observability is another piece where teams are stitching together multiple tools and running observability tools in the clusters that they're meant to be observing.

Bart Farrell: How does networking work when applications running on-prem need to communicate with services running in the cloud?

Raglin Anthony: So when applications on-premises need to communicate on cloud, they need a network. Path that can bridge these two otherwise isolated environments. And the simplest way is going to be traffic routed over the public internet using HTTPS encryption. But then you're at the liberty of the internet variability when you go down that path. A more private and secure option could be a VPN solution like AWS Site-to-Site VPN. And that will create a seamless network between your in your on-premises environment and your VPC in cloud. And then for like a high throughput, low latency, highly reliable network connection, you could use AWS Direct Connect. And that's a physical cable that connects your data center and AWS. It bypasses the Internet completely. It's suitable for like production workloads. It's also suitable for compliance related workloads. Either which way you go, whichever path you choose, you've got to still figure out DNS. Is always a requirement to make sure that on-premises is able to resolve those cloud host names. So that's pretty much how the networking would work.

Bart Farrell: Now, AWS launched EKS Hybrid Nodes. So how does it address the above operational challenges across multiple environments?

Raglin Anthony: So EKS Hybrid Nodes, I would say, directly addresses pretty much most of the challenges, if not all, that we've just spoken about. The first one being like control plane in terms of being able to do the scaling, the security, the hardening of that entire environment, patching and upgrades. That's all taken care of by AWS. So that burden of control plane management is completely gone. The second piece is identity and access management. So EKS hybrid nodes would integrate natively into IAM from AWS and would leverage things like EKS pod identity for security in terms of accessing AWS services. Or you could use IAM roles for service accounts as well. In terms of the networking so it'll use the existing networking structure if you're using vpn or if you're using direct connect so you're not having to rewire your entire network in order to access aws services you're using an already configured environment and just kind of relying on that configuration additionally your nodes in itself stay within the network in your environment it's on premises your pods have their own network so they're able to communicate with each other so you're you know got full control of your nodes and you can leverage tools like nodeadm which is a CLI tool from AWS to handle the upgrades of those nodes. So it simplifies the management in terms of the differences between your control plane and your data plane, while still giving you control of those assets on-premises.

Bart Farrell: Kubernetes turned 10 about two years ago. What should we expect in the next 10 years?

Raglin Anthony: So in my opinion, I think more autonomous clusters self-healing. I think we're seeing a lot of headway that's going on with agents going on in clusters. MCP tools that's also coming out through. And I think we're going to get to a point where cluster operations will become a lot simpler, where you'll be able to create a cluster and hopefully at some point it can automatically upgrade itself and self-heal. I think that's going to be a big focus over the next 10 years.

Bart Farrell: And Ragz, what's next for you?

Raglin Anthony: Well, I hope to be one of the contributors to those autonomous clusters. I spend a lot of time in the media and entertainment space, so the other thing that I'm focusing on there is trying to leverage like a few watts. To start creating like editing workstations out in the cloud. So that's something that I'm putting a lot of focus over the next couple of months.

Bart Farrell: And if people want to get in touch with you, what's the best way to do that?

Raglin Anthony: I think the simplest way would be on LinkedIn. I go by Ragz, but you can find me on LinkedIn as Raglin Anthony.

Subscribe to KubeFM Weekly

Get the latest Kubernetes videos delivered to your inbox every week.

or subscribe via