Why PostgreSQL operators are the future of database management in Kubernetes

Bart: Who are you? Where do you work, and what is your role? The speaker is Gabriele Bartolini (works for EDB).

Gabriele: Hi, my name is Gabriele Bartolini, and I work for EDB. I'm the VP and Chief Architect of Kubernetes, and my role is to bring Postgres to Kubernetes. I'm also a maintainer of an open-source project called CloudNativePG and a Data on Kubernetes ambassador.

Bart: So, what are three Kubernetes emerging trends or topics that you're keeping an eye on?

Gabriele: I am a database person and a Postgres contributor. My interest is in bringing Postgres in Kubernetes to the same levels it had outside. Another area of interest is storage, as I am always looking for new storage solutions in Kubernetes, given that storage is a critical component in databases. Additionally, I am interested in how databases are consumed, such as using AI. My goal is to make the database, for instance, by utilizing PG vector or creating a vector database, consumable through a generic API for drag-and-drop applications.

Bart: One of our guests, David, strongly advised against DIY solutions for running Postgres on Kubernetes, calling them a professional disservice that will lead to technical debt and operational challenges. Have you ever attempted to build a DIY solution for running Postgres on Kubernetes? If so, what challenges did you face and do you agree with the author's statement?

Gabriele: David is a wise guy. I totally agree with him. Nowadays, you cannot imagine running a database like Postgres as a container, such as a pod or a deployment. The do-it-yourself solution is not applicable for a database like Postgres. The challenge is managing log files, monitoring, observability, backup, and recovery when trying to implement this approach. These are all things that involve a lift and shift approach from VM and bare metal into Kubernetes, which must be done within Operator Capability Level.

Bart: David was critical of the "start small" approach for Postgres on Kubernetes, citing examples where it leads to governance issues and technical debt within months. What has been your experience with starting small for Postgres on Kubernetes? Have you encountered similar governance and technical debt issues, or have you found a way to make this approach work successfully?

Gabriele: The 'start small' approach is nowadays unthinkable. Operators like CloudNativePG help bypass challenges immediately. My suggestion is to use an operator. There is no difference in performance when using Kubernetes with Postgres inside, compared to running it outside in VMs or bare metal. There are no limitations. I agree that we shouldn't start small; instead, we should consider Postgres as a first-class citizen in Kubernetes with the aid of an operator like CloudNativePG.

Bart: David was skeptical that even the best Postgres operators in 2024 would be ready for full autonomous operation, emphasizing the need for advanced feature support and some level of human oversight. Do you share the author's skepticism? Have you encountered any operators that have already reached an advanced level of feature support, including security, extensibility, connection handling, and observability, while still requiring some human operation and oversight, considering the Operator Capability Level?

Gabriele: Again, he's right. As a maintainer and co-founder of CloudNativePG, I can say that CloudNativePG only starts at Operator Capability Level 5, according to the operator capability level framework. This means we always need human intervention, and there's still a lot to improve. For example, we are missing major upgrades and want to perform them in a declarative way, using Declarative PostgreSQL Management. Currently, human intervention is required to create a declarative solution for migrating from PostgreSQL 15 to PostgreSQL 18. However, we aim to make this process declarative with in-place upgrades. Another area for improvement is the scaling of storage or resources. Additionally, we need to work on extensibility. PostgreSQL is famous for its extensibility with extensions like PG vector, TimescaleDB, and PostGIS. However, our operator has a significant limitation due to the usability of containers, which requires us to preload all extensions inside the base image. To address this, we are working closely with Kubernetes and Postgres to improve the way Postgres manages extensions, allowing us to load them on the fly, starting from PostgreSQL 18. We have a patch for Postgres to improve this, which we are trying to implement. I agree with him that DBAs will not lose their jobs with Kubernetes; instead, their role will be elevated. They will be able to focus on more exciting tasks and work closely with development teams as part of a multidisciplinary team that works across the organization.

Bart: Kubernetes turned 10 years old this year. What should we expect in the next 10 years to come?

Gabriele: I think I would expect to have more users coming. In 10 years' time, I think we will be talking about the next Kubernetes. Kubernetes will be the standard, and we will not be discussing how to run Postgres in Kubernetes anymore. It will be the de facto standard, like VMs are now. And I'm sure we will be talking about the next technology that will disrupt the industry.

Bart: What's next for you?

Gabriele: Next, I will keep improving CloudNativePG, APG, and running Postgres in Kubernetes. I have just finished the first phase of bringing Postgres into Kubernetes the way we wanted. Now it's time to work closely with DBAs all over the world who know Postgres, because we need their expertise and experience inside Kubernetes to facilitate and grow the adoption of Postgres. To get in touch with me, Gabriele Bartolini is the best way, or you can join the Cloud Native PG Slack channel or the Data on Kubernetes (DoK) channel and reach out to me there.