Bart Farrell: So welcome to KubeFM. First of all, who are you, what's your role, and where do you work?
Fernando Harris: Okay, thank you for having me. My name is Fernando Harris. I am a Portuguese Mozambican living in Spain with a beautiful family. I work for Oracle Cloud. And basically what I do in Oracle Cloud is I support our partners and customers, advising on best practices, running cloud native and Kubernetes workloads in Oracle Cloud. Basically that.
Bart Farrell: How should teams connect Kubernetes resource usage with application, transaction, and database behavior before deciding something is genuinely over-provisioned?
Fernando Harris: Well, I believe that over-provisioning something, it's not necessarily immediately negative, right? I mean, we need to analyze it properly, right? So when something is over-provisioned, we don't really feel the pain immediately, in theory, because we're not talking about resources being scarce, it's just the opposite. So in theory, from an infrastructure point of view, everything should be there, at least to help us provide what we consider to be the valid functional responses of our applications, of our pods. As long as our pods, they don't have any functional errors, they don't have any applicational problems or logical problems, at least from the resources point of view, everything should be there. So our customers should be happy. Okay. This is basically an easy way to follow kind of operational happy path. Of course, things are not like that. Over-provisioning can be a strong symptom of lack of efficiency and a symptom as well that there is room for optimization. And I mean, we don't know for sure, but there might be some waste, right? And waste is a very important term here when we are discussing these cost optimization discussions and considerations. there is a hypothesis to find some waste, not a fact. To become a fact, we need to analyze it a bit deeper and to get to know the workload a bit better. First, in terms of what is waste, or what is waste removal, is actually the basic primitive of philosophies like Lean, which eventually influences Agile and all the DevOps stuff. In the past, over-provisioning was normal. in the beginning of a project because you basically procure infrastructure for every new big project or major new development. And we still call it capacity planning. A lot of people are doing it perfectly, it's valid, specifically if you run your own data center. It implies bigger investment, it implies more capital in the beginning, all the capex stuff. At some point in time, you could face under-provisioning and you would then buy more capacity. right? And repeat the old procurement process all over again, all the bureaucracies, all the steps, right? So cloud changed that and brought all the facts. So that's why this question around over-provisioning is also more important, more relevant, and it's more dynamic, right? So less money in the beginning or less capital, but there's a big caveat: unexpected bills at the end of the month, right? So it's one of the main reasons to migrate workloads, believe me. So cloud with the broad FinOps. And FinOps itself, as an organizational culture, should also help removing waste. So we already are talking about removing waste. So from a practical point of view, these discussions should be straightforward. And it's not. And why it isn't? Because sometimes over-provisioning is basically leaving idle capacity. And that idle capacity can be there because of a specific valid reason. Maybe to guarantee the capacity or the means to deliver business value according to some predefined SLOs from an application point of view or from the business point of view. So my SLO should be connected with a business function that will deliver some sort of value for the customer. So we need to have the capacity as Kubernetes infrastructure owners or Kubernetes platforms owners or platform owners working with Kubernetes. to understand this as well, to understand this boundary of the transaction as well as you mentioned in the question. Like the transaction is an agreement between parties and it's a commitment, some sort of commitment, right? So, and it has content, it has data, it has ID, and it should be as well in parity with the business function, right? So, I like to give an example, which is a very typical example. Every company selling a product online will have some sort of payments feature, right? So, payments or checkouts features. These payment feature will have other technical dependencies, other financial intermediaries, APIs to consume, bank APIs, of course, database or more than a database. And these kind of dependency or dependency technical functions, let's call it like that, will also need to work together to deliver something useful on time for us and more specifically for the customer. So every time I'll register a payment, I will register the transaction. Just to sum up, teams working with Kubernetes, they will need to understand these boundaries. They will need to connect resource usage with this concept of business flow, which is going to feed the business functions to guarantee the delivery of business value. And its criticality. Here again, criticality can be defined by the SLO itself. And these SLO can help define things like non-function requirements, availability, resiliency. which we need to implement as architects to deliver value, right? So this is all very important for the team managers in Kubernetes. They need to connect to all the stages of the business flow. And they will have also proper correspondence from a Kubernetes point of view because we have the concept of Kubernetes workload, right? The pods which are involved in the process and in the transaction and dependent eventually also of database transaction. So I need to see time, I need to see timestamps, I need to see the beginning and the end of a business flow. and connect my business SLOs with it. Things such as how many successful payments I have, how many successful checkouts I have, or how many persons abandon the payment process. This is all useful, valid information, relevant for the delivery of business value. Then I connect that info with the application again. The application things such as the number of requests per second, the number of errors, latency, retries. Then I can go on a database level. From a transaction point of view, I need to get information about how much time it takes to run a query, how many queries are under a specific amount of time, how many connection pools, how many connections are open, how many I can still open, right? And all these perspectives, right? I mean, it's a holistic perspective, will help me adjust the way my Kubernetes team is managing CPU and memory requests, the limits, the real usage, the throttling from CPU. out of memory kills, a number of replicas, all that information which we expect to receive from a Kubernetes environment. So in summary, from a business point of view, I validate useful outcomes for the customer on time. From an application point of view, I validate service quality and concurrency. From a database point of view, I validate eventual transaction bottlenecks. And from a Kubernetes point of view or infrastructure of Kubernetes point of view, You have a reserved capacity versus actual usage, right? If I have over-provisioning, a pattern of over-provisioning, it's only genuine, as you asked, if by reducing the number of replicas, for instance, I can still guarantee a useful outcome, business value being delivered. If when I reduce the number of replicas, I negatively impact my outcome, then the situation is different. That over-provisioning is there for a reason. Right. So maybe. I have idle resources on my cluster because I want to protect an upstream system or a downstream client. It's not just about saving money by shutting down nodes or reducing the number of replicas. It's about understanding a real impact on upstream and downstream systems and eventually on the quality of service and delivery of value or time to our end customers. Basically, this is the big perspective in terms of giving the Kubernetes teams this sort of responsibility and holistic perspective in order to decide.
Bart Farrell: And Fernando, how do you distinguish we need more compute from auto-scaling is just making an underlying dependency problem more expensive?
Fernando Harris: This is a typical problem with cloud and Kubernetes in particular, but cloud in general, right? Because any service that scales up and out with resources on demand. if or when there is something wrong with the system itself, that doesn't mean that scaling up or scaling out will fix the problem, right? And a lot of people start using cloud and start using Kubernetes with this wrong perception, right? So the system is slow, so I'm going to add more pods, right? And this is not always true. For instance, if we have, with the figurative example, right? If we have 10 pods executing 100 requests per second and my database queries are taking around 100 milliseconds. So if I add more compute, 10 more pods, 10 more replicas, right? And if I can still guarantee that the database queries are still around the same time, 100 milliseconds, and I have more requests per second coming from the pods, but the database performance is stable and consistent, then I can say confidently that I'm adding more compute and the system is not responding negatively as a whole. So I didn't identify any sort of database degradation and I can continue to deliver value to my customers. So my system is always stable. This is where it makes sense, but the problem is when I had these same 10 pods and let's say that I see a degradation of the database. I don't know, 50% degradation. The database, 150 milliseconds to give an answer and a proper response to a query. So increasing the number of replicas in this case is not necessarily helping. We are actually facing the what I mentioned in the previous question, a bottleneck on a database level. It's a dependency bottleneck. And extra pods are actually creating more concurrent database calls, more waiting, more timeouts, more retries. HPA will see busy pods and it will try to keep scaling pods, right? And the end result for the business can be worse. The end result for an SLO of an application can be worse. And we'll eventually create a bigger problem or a more expensive problem to solve. So we'll need probably to find workarounds, we need to throttle, to filter, to reroute, to consider using cache to divert specific requests to cache, to not hit directly the database so often, so frequently, all those sort of tools or approaches, or perhaps we need to increase the database resources as well. And this is the same rationale if it's not the database, but another application, not necessarily database. If there is a dependency between two applications, the same rationale applies. We need to probably fix the dependency first or the nature of the relationship with that dependency. This is the type of rationale which normally domain-driven design practitioners and architects do very well. They anticipate the relation or the nature of the relationship between two domains, two systems, two applications, two pods, one application and one database. And they can try to predict this type of behavior. try to plan for it, right? So it's not clear if we simply add more compute, we are not going to be more successful from a business point of view or an operational point of view. That's it.
Bart Farrell: How close can observability get us to understanding the actual cost of serving a transaction or workload rather than just cluster level resource consumption?
Fernando Harris: Basically, we're asking if we need more granularity in terms of understanding these costs, right? Like not just understanding from cluster-level resource consumption, right? If we need more granularity, if we need more properties to understand it, how can observability help us, right? So, well, if we think about everything I just said, I mean, the two previous questions, this question is kind of the cherry on top of the cake for this exercise. I mean, what is observability about, right? What is observability? If I want to say it in a figurative way, it's just I need to be aware of everything occurring in my system, right? All layers of my system, business flows, applications, databases, the infrastructure, the infrastructure for Kubernetes, storage, memory, CPUs, whatever, right? Again, as a solution architect, in my experience, I need to guarantee or to help protect business value delivery on time for my partner, for my customer. This is my why. That's the reason I'm there. I have always this perspective of business value delivery on time. Also, as a solution architect, I need to know how to do that. So, I mean, the methods I'm going to choose, the patterns, the architectural patterns, the tools I'm going to use. So, observability principles, not necessarily the tools from the kickoff, but the principles themselves will show me kind of the gaps between my why, and my how. So what I mean is the tools I have in place to implement are going to help me deliver the business value on time. And also the gap between these and the unpredictability of my business, the complexity of my business, the real usage of the system. Maybe we planned for one of the 1,000 users and we were so successful that we now need to deal with one million users. And if there are gaps, then there is work to do from a solution architecture point of view. And observability principles and tools are key here. In my opinion, in particular, the utilization of traces, logs and monitoring, of course, all the different aspects and objects of observability are important, but tracing in particular can help us identify and map and follow a specific business flow. So with a unique ID, its instances, its transactions, its boundaries, they go through different domains and different systems, and yet they keep the same ID, as I said. So I can kind of cut a slice of my organization from a business point of view and try to understand it. Right. So having. the trace as a reference, an atomic reference of what is happening with the specific transaction, it's really important because I can then develop monitoring strategies to help me focus on the database behavior or performance with the database metrics or, as I said before, things such as query duration, queries, number of queries, connection waits, IOPS, locks, etc. Right? So, and then I can focus as well on the different layers, on the application layer, with the application behavior, which is based on standards like open telemetry, and then using tools like Grafana that have access to the number of requests, HTTP requests, the latency, the errors, the nature of the errors, number of retries. And then from an infrastructure point of view as well, with Kubernetes and Prometheus, for instance, to have access to CPU, memories, GPUs, if I'm running AI workloads, storage, a bunch of things I need to keep an eye on, right? all these metrics together with open cost and the open cost, the way I consume APIs from different cloud providers with open cost, I can then eventually allocate these Kubernetes infrastructure costs by namespace, workload, the typical stuff like pods and services and labels. And again, with Grafana, I can correlate these allocations with primitives, metrics, and of course, the open telemetry traces and other business metrics. Just to conclude, I can consolidate or reconcile all this information. reconcile, I think is kind of the financial technical term, against the bills emitted by my cloud provider and having a proper granularity of what I'm consuming and how I'm consuming. Ideally, and I emphasize here again the trace importance, ideally I want to be able to say something like in order to have 2,000 payments or checkouts like the example I gave before, I don't know, 2,000 checkouts per second, right? I need to run my payment service, which will cost 10 euros for APIs, 10 euros for workloads, 5 euros for the database, 5 euros for cache. I need to have this type of granularity. This is the perfect marriage between open telemetry, the capability of open telemetry, and open cost to help us build this sort of financial perspective, which I believe is fundamental for future Kubernetes cost optimization.
Bart Farrell: Fernando, what's next for you?
Fernando Harris: I think we are right now in the moment of a lot of really interesting things happening. It's interesting, at least for this discussion here, to understand what will be the role of AI in terms of cost optimization, cost consolidation. There are so many areas interesting to study right now. As a solution architect, next for me is to continue being close to what the business reality is, right? Which is kind of agnostic to the technology, to AI, etc. Business value, understanding the organizational complexity, the architectural complexity, the domain complexity, it's still what is next for me. So I tend to try to consolidate my canonical knowledge in order to understand the real impact from these modern new tools, mostly based on AI, to help improve the Kubernetes ecosystem and the operations related to Kubernetes.
Bart Farrell: And if people want to get in touch with you, what's the best way to do that?
Fernando Harris: LinkedIn. I think LinkedIn is the only social network I keep, so I normally respond under 24 hours. So it'd be a pleasure to continue to discuss these topics. Also through the CNCF Malaga, right? We run the CNCF Malaga, the Cloud Native Computing Foundation meetup in Malaga. I'm also accessible through its channels.
Bart Farrell: Thanks so much for joining today, and can't wait to interact with you in the future. Best of luck, Fernando. Cheers.
Fernando Harris: Thank you.
