Bart Farrell: Welcome to KubeFM. For folks who don't know you, who are you, what's your role and where do you work?
Koshin Verberne: Yes, I'm Koshin Verberne. I work at Rabobank here in Utrecht. It's actually where I was born and raised. I'm a container platform engineer at the bank. And in my previous roles, I've also worked within the platform engineering space for the last five to six years. That's who I am.
Bart Farrell: And can you give us a bit of context on your role at the bank and what the Kubernetes landscape looks like there? What kind of Kubernetes environment are you helping build and operate?
Koshin Verberne: Yes. So we started a Kubernetes platform team two to three years ago on all the landing zones where we are hosting our applications and banking systems to offer our consumers the Kubernetes platform experience. And the kinds of environments we are hosting are a mixture of public clouds such as on Azure, AWS and also on-premises with OpenShift that has already been running for a couple of years there, and potentially future environments where we might need to learn, such as a European provider, but that we still need to see. And how it's operated? We have two operating models. So for on-prem OpenShift, we offer our tenants a namespace as a service. So you land on the same cluster as all the other tenants, but you get a unique namespace allocated to you. And in the clouds, because of scalability and isolation, we can offer on an account level. We there offer a cluster as a service that is dedicated to each tenant: their own Kubernetes API, pretty much.
Bart Farrell: And in a talk that you gave recently at KubeCon, you walked through the evolution from namespace-based multi-tenancy towards stronger isolation with accounts, networks, and cluster as a service. So what problem were you trying to solve and what changed your thinking about how Kubernetes should be consumed in a bank?
Koshin Verberne: Yes, so the problem we were trying to solve there is having the right balance between isolation for our tenants and the number of responsibilities we, as a platform team, would need to take ownership of to offer the platform in a secure and a compliant manner. For example, hosting it in namespaces in, let's say, AWS (because I have most of my experience there): doing this in a namespace way would imply that you're hosting it in one central AWS account where all your tenants land in and that would basically be in the same network. But the integrations that you might want to offer to tenants to, for example, access a database, access file storage, such as in S3, you would need to really have strong guardrails or a separation between what identities tenants can use and how to set up the right access to resources that only should be consumed by that particular tenant if you're going to host them all within the same Kubernetes cluster. And unfortunately, RBAC has some limitations there: what you can and cannot restrict people from doing. So you would need to take a lot of time and effort there to restrict permissions for what teams are and are not allowed to do in a cluster. Also within the limited time that we have, of course, and resources to build the platform, you can imagine doing it with, let's say, a small pizza team. This could grow easily very complicated. So what we wanted to do was be able to handle the number of workloads that would come our way and have as little friction or limitation on the side of Kubernetes as possible with integrating with the cloud services that these might want to consume. If you want to use your S3 or DynamoDB tables, you can do so because the isolation is already made there on an account level, so you are guaranteed to have the best security for your workload. So what we really want to be able to do is to offer this platform to any kind of workload. If you have high confidentiality requirements or if you're just hosting a simple web application, you can still land on our platform. So that is what really helps us with offering these clusters to any kind of tenant now and in the future. So upfront, the onboarding would maybe take a little more time than when you have it all hosted on a central cluster. Creating namespaces is done in seconds. But creating a cluster, that could consume quite some time still from running a Terraform pipeline and registering the identities and doing the onboarding within the central IDP. But once that's there, you are guaranteed to have a strong foundation and guardrails in place to give you a secure and compliant way of working at the bank. So that is what was our priority at the time with going for this architectural route. And perhaps in the future, when there is more demand for more flexible, let's say, lightweight workloads that we would want to cater to, then of course we can look again at the architecture where we say, okay, certain workloads: it would be overkill to have your own Kubernetes control plane. So we might want to have that separately hosted for namespaces. So that could also be a decision we want to go for. But for now, we have a strong baseline there, a strong foundation where we can offer it to anyone in the bank. And especially if you see the latest developments around AI and isolation requirements there in Kubernetes, this is also helping us to meet the requirements of these more novel developments going on within the bank or potentially other banks, to serve as a cluster to teams who require it for their AI-based workloads or maybe agents as well, still with strong isolation in place as a foundation. So that really pays off if you have an architecture such as what we have followed.
Bart Farrell: And one of the interesting trade-offs in your story is that stronger isolation can mean more infrastructure and more cost. How did you decide where the additional security and isolation were worth the extra complexity or cloud spend?
Koshin Verberne: Yes, so for the cloud spend, we of course try to optimize how workloads are provisioned or which resource requirements tenants might have. So we do try to challenge them on this. For example, at the Rabobank, we have a dedicated FinOps team who also looks at this data: how many resources do you consume and what was actually required or needed for applications? So all these telemetry data we gather, and they are made available to the FinOps team, so they also can play a role there to help further optimize workloads for our tenants. And in addition, we are able to scale the nodes for clusters in EKS using a nice service called Auto Mode. So there you have basically the capabilities of a Karpenter there to help determine the right node size for you. So in that way, we are also able to do rightsizing much easier, much faster than having to determine up front: which type of compute models do I offer or do I do t-shirt sizing? We fortunately don't have to consider these decisions or designs up front because we have leveraged the Karpenter for this. So that's really great. But cost-wise, of course, having it all on a central cluster or doing a namespace as a service would save you a lot of cost. But then referring back to the security requirements we have, and having a strong isolation for some of the most sensitive workloads that you could have as a bank, we viewed this as a more heavy weighing, having weighing stance on being able to meet their requirements also relatively easily this way, without needing to build all types of layers on top of vanilla Kubernetes, let's say. For example, not having to do complex deployments for certain types of tenants or where their requirements might differ from others. And where we are able to say, for example, with Kyverno, you have the freedom you need to provision your workloads and your namespaces that you might need. Because we also host a lot of commercial off-the-shelf applications. Not everything is built in-house, of course, at a bank. So also for those environments, they would need to have their own cluster. So in that way, yes, we do consider a cost to be an important factor, but that would never, or should never, outweigh the security and the robustness that we expect to be able to offer to our tenants. So that's of course a balancing act that you need to make at a bank.
Bart Farrell: So looking back at the journey, what are the biggest lessons for organizations operating Kubernetes in a regulated environment? And what do you think teams need to get right from the beginning around isolation, security, governance, and developer self-service?
Koshin Verberne: Yes, that is where architecture plays a big role in how you decide your operating model. And architecture for Kubernetes involves a lot more than just the cluster in itself because any team can create an EKS cluster or AKS cluster in Azure with the click of a button. But that's not where we add the value. The value there comes with the integrations we offer. Also with strong identity and access management, certificate management, DNS, load balancing, and what I just mentioned about Kyverno, that is also part of governance. That plays a big part. That's the added value you would want to offer and enable for your tenants to leverage it as easily as possible. You should try to think about making also your platform a bit interesting, let's say, to want to onboard. Because in the past, there were a lot of teams or DevOps engineers who were running their own Kubernetes clusters, especially in the cloud, because it's so simple within your own account or subscription to request one. And we didn't have a lot of standards around Kubernetes, let's say, several years ago. So there, what we are able to offer or to enable as a platform team is to worry less about the underlying infrastructure and foundations of Kubernetes and enable the tenant to focus more on delivering business value and work on the application side of things. So wherever we eat in the architecture can choose: how do we offer this, for example, managed services within Kubernetes? Whether we talk about certificate management or secrets management, managed GitOps, for instance. It's very simple or relatively easy to set up your own Argo CD, but how do you make it in a way that it's also compliant and that you're allowed to run it in production at a bank? That would involve, for example, also identity and access management, change management. All these kinds of decisions that you would want to think about up front for your tenants to help solve those kinds of challenges that you can already see. If you're going to do A, then you're also expected to follow these requirements that we have in the bank. So that is where we really offer the value there. And with regards to the operating model, that's what I've explained already. It helps to give a strong foundation: you can also trust us with hosting your application workload. It's also showing to our tenants: some of them are relatively small. It's just simple websites or hosting something small for the bank or maybe something new. But we also would want to show to the bigger tenants who might be offering or hosting mortgage systems or real actual banking or payment infrastructure that what we offer is robust and resilient and can help them to more easily do the hosting part of this if they want to land on Kubernetes. So that's everything you have to consider, let's say, as a platform team. It's really trying to take away the complexity so that tenants can have more focus on what really matters to a bank.
Bart Farrell: And just as a bit of a bonus question, since we're touching on things that really matter to a bank, things around ownership, we talked about using third-party tools previously and things that perhaps have to be done in-house. But a very big topic right now for financial institutions is digital sovereignty. And so it's becoming a much bigger concern, particularly for European banks. How does that influence the way you think about Kubernetes architecture, cloud providers, data location, and avoiding too much dependence on a single vendor?
Koshin Verberne: Yes, that's a good question. So trying to avoid too much of a dependency on a vendor is where we, for example, look at certain infrastructure components in the way that we host these. So for example, we have Argo CD managing deployments. You can host that anywhere on any type of infrastructure. Istio, the same. You could host that anywhere. It's just the flavors of how you host it or how you set up the controllers for managing resources or infrastructure in a cloud or in another environment. We need to look at: how would it scale well or how would we be able to have, let's say, a rich offering of features that we require to make also the security at a good enough level? Because that's always the most important part. Then certain dependencies you will always have. Because using, for example, certificates in Amazon, we would rather use ACM than host our own certificate store in the cluster. We would want to also make our lives easier that way. But in a future scenario, if you're going to do that with cert-manager, for instance, and we think we have the amount of capacity or demand for it to host it ourselves, then we know those capabilities are out there in Kubernetes. It's just a matter of: How do you design it? Who takes the ownership of that or the responsibilities to do that part of the service offering? So in that way, Kubernetes offers a lot of opportunities that we could pursue as a bank. But of course, we don't want to be semi providers or infrastructure hosting in-house with regards to having to provision all these solutions yourself. Because you build it and you run it yourself within cloud native or having multiple vendors doing a little bit of your infrastructure. So also on that front, architecture-wise, we always need to carefully consider which parts do we feel like should be owned by ourselves, when it comes to hosting Kubernetes or which parts we feel like, in this case, a cloud provider or public cloud provider would do it just much better than ourselves. Because we're also not the networking experts or the best on security by having, I don't know, our own key store. You could do that, of course, but then you would also need to have dedicated engineering teams for this, which are out there, but they host for their own tenants. Or if you would want to host certain legacy systems, there's a lot of design work that you would need to have in place. And also at Rabobank, we don't put all our eggs in one basket. So Kubernetes is one of the technologies that we offer or that we would want to run our applications on. But you have other solutions such as Cloud Foundry, which are also being used. I'm not sure if you are familiar with it. So for people who might think: we're going to run everything on containers. That's also not totally the way we would want to go. Because running everything in containers or in Kubernetes, that in itself has also its complexity. So solutions that would better fit on other platforms, or maybe some other low-code platforms, let's say, would fit those types of workloads. So Rabobank is very varied in that aspect. That is always what we should also consider, let's say, whether certain applications are a good fit for Kubernetes or whether they might need to be hosted elsewhere. There's always a balancing act in that sense when you are really running at scale in an enterprise.
Bart Farrell: Challenges that people have to keep in mind. No, I really hear you on that. Well, Koshin, thank you so much for sharing your time and knowledge with us today. I know this is going to be very beneficial for people that have been checking out the report that we released about Bank-Grade Kubernetes. Wish you nothing but the best of luck in the coming months with the challenges that the bank is facing around these topics of digital sovereignty, ownership. The guardrails have to be in place while trying to give people the Kubernetes-style experience that you mentioned, which involves a fair amount of freedom. Best of luck, and I look forward to speaking to you again in the future. Take care. Cheers.
Koshin Verberne: Thank you. Bye.
