Bart Farrell: So welcome to KubeFM. For people who don't know you, who are you, what's your role, and where do you work?
Ram Vennam: I'm Ram Vennam. I'm one of the solutions architects here at Solo.io. I've been here for over five years. I've been working in the Istio community since almost the beginning of the Istio project. But at Solo.io, I mostly help our large organizations adopt Service Mesh, API gateways, and now AI gateways. So I'm happy to be here.
Bart Farrell: So Ram, in a regulated multi-cluster environment, how should teams enforce workload identity and east-west access without turning every application change into a security ticket?
Ram Vennam: that's a good question. For multi-cluster, we're seeing an absolute rise of adoption of large enterprises adopting multi-cluster for various reasons, Across region, across cloud providers, and even spanning on-prem and public cloud. So there we definitely see, a rise of multi-cluster. When we're thinking about things like workload identity, stuff like workload identity, authentication, authorization, certificates, all this stuff, it really needs to be outside of the application scope. So it can't be in the application, it can't be in the source code, the container. It should really not even be of developers' concern. So this is all now, the platform team's concern. So, regardless of where the workload gets deployed, and the keyword is where, the location of the workload should not matter. And what this means is, if you're writing policies based on things like network principles, like IP addresses, insiders and firewalls, that's where we see change resulting in outages, So now we need to move to a point where we're relying on the platform to provide location-agnostic workload identity. And then the security teams have the guardrails in place so that their identity and authorization concepts know where their clusters are spinning across the various different boundaries. And then from the application team's concern, have to provide the platform team some metadata that says, these are the services that I need access to, or my application needs access to, and then these other applications are allowed to consume, And then now the platform team will convert that in their pipeline to the right authorization policies that get applied, based on the primitives that workload identity gives us directly from the cluster.
Bart Farrell: When failover crosses clusters or regions, how do you automate resilience without accidentally violating data residency, trust boundary, or access control requirements?
Ram Vennam: good question. When we're thinking about failover, typically that is a resiliency concern. It's almost like a preference where the platform engineering team says that I prefer when these workloads in this region fail, I want you to failover to the closest instance. And then they might declare that this is where I want you to failover. And then if that doesn't work, then failover to here. But that's not, that should not be confused with data residency. Data residency is not failover. This is a security constraint, So again, we need to go back to like the security first principles and apply things one step at a time. So by default, even in a multi-cluster environment, services should by default be scoped to a local cluster, Like, just your local, the cluster that you're running on. If a service is available in multiple clusters, then you need to declare exactly which subset of your broader mesh you want to scope things to. we have this concept of segments that allows you to group together sets of clusters that you then have this additional scoping primitives to. And then if it's a truly globally available service that you want it to be across all of your clusters, then you can declare that service as global. So this tiered approach is very important from a security posture. The second key thing is being able to write policy, like your authorization policy, not just based on the workload's name or their service account. but also based on the attributes of that workload. So what that means is from an application developer's perspective, when they apply an application to Kubernetes, In their pod spec, they're able to declare custom attributes that define what their application is. Like this is a PCI application. This application can only run in the EU region or be called by other workloads in the EU region. These are attributes that they toss on their pod. And then when you write an authorization policy, like a service mesh authorization policy, you have to leverage those. So those claims need to go into the certificate, into the workload identity, the SPIFFE cert. And then when you write policies, you can leverage those. That's what prevents things from accidentally going across region. Failing over is just like a routing and a service discovery concept. Like workload attributes is now a security concept.
Bart Farrell: If networking security policies are managed declaratively, what needs to be logged, versioned and verified so you can prove not just what the policy said, but what was actually enforced?
Ram Vennam: like you said, what's in your source of truth, like these are your policies, that's your intent. And that's how you want things to look, operate and be enforced. And that's great. that way you can track the history, you can have the approval-based workflows and everything. But that is the very first step in the pipeline. When you're thinking about what actually gets applied, then you have to look at things like your Kubernetes admission webhook to see if it's actually applying every single one of those policies. And then if it doesn't get applied, are you being alerted so that something was rejected by the admission controller? And then next, if it passes that point and it actually gets applied on the cluster, maybe it failed to be accepted by the control plane. So every one of these resources has a status, so the control plane can typically read this resource and apply the status to the resource itself. So that status absolutely needs to be monitored for all of your resources. Now I've seen cases where the control plane accepts the configuration and then the status looks good, but then internally it fails due to some duplication or some other race condition. and it never actually makes it to the data plane. So the data plane also needs to be verified. So the data plane typically exposes some sort of config dump where you can get a rough snapshot on exactly what is being applied that the data plane is actually going to enforce. So in a highly regulated environment, that is worth the extra effort for you to have some checks in place to check the data plane itself to see if your configuration is being applied. And then finally, when the data plane actually enforces the policy, it should write an audit log. It should write a log line that says, this workload with this SPIFFE ID tried to access this resource and it was rejected. Obviously, all the basic things like, when did this happen? How often is it happening? These are all things that we need to have a dashboard for so that we can observe and detect this early on.
Bart Farrell: So, Ram. It's been almost a year since the Ingress NGINX retirement was announced. So in bank-grade environments where you care about stability, auditability, change control, and long-term support, how is the move toward Gateway API actually going? Is it mature enough today for regulated platform teams to standardize on, or are there still important gaps?
Ram Vennam: The Gateway API, in my opinion, it's completely mature and it's being adopted by multiple different technologies and it's battle-tested right so Istio Ingress Gateway implements Gateway API, KGateway implements Gateway API, AgentGateway implements AgentGateway API. So there's multiple projects that have put it through its paces and they've extended it to cover any gaps that people have traditionally implemented with, let's say, like an NGINX controller. So there's various policies that attach to the Gateway API to give you that same functionality. Everybody that we talk to, we give them a clear path to migrate to Gateway API. Regardless of what implementation of the Gateway API that they choose, we haven't seen any gaps yet. So we should 100% be adopting it.
Bart Farrell: What's next for you?
Ram Vennam: Next is agentic workloads. I think up until now, we've focused a lot on securing, observing normal HTTP REST-based traffic. We need to do the same for agentic workloads. The patterns are the same. What exactly we are observing or enforcing is a little bit different with the agentic traffic. We went from request rate limiting to token rate limiting. We're now writing policies based on the prompts. For MCP, we're doing things like tool filtering, like down to the tool level and being able to do multiplexing. So these are all things that are squarely in the space of networking. So the proxy that we use to achieve those features is a little bit different. So my focus is moving from just using Envoy as the main proxy to accomplish this to now also focusing on Agent Gateway as the proxy for all this agentic traffic. But yes, right now we see both Envoy as well as Agent Gateway.
Bart Farrell: And Ram, if people want to get in touch with you, what's the best way to do that?
Ram Vennam: LinkedIn. You can find me on RamVennam or my email is ram@solo.io I'd love to answer any follow-up questions that people have.
Bart Farrell: Great. Well, thank you so much for sharing your time and knowledge with us today. And look forward to speaking to you soon. Take care.
Ram Vennam: Awesome. Thank you.
