Moving cloud operations to a Kubernetes operator

Bart: Can you run stateful workloads on Kubernetes and not die in the process? Difficult question people have been asking for a while and something I can proudly say that I actually know a fair amount about. That being said, what are the challenges that people are faced when they decide to run databases, stateful workloads on Kubernetes? They have to figure out if they're going to be using an operator. And if that operator is something they're going to buy, if they're going to build, what are the necessary tools to get that in there? What's the difference between an operator and a controller? These are all questions that folks like Steven Sklar, who's an engineer at QuestDB, have to address. Although Steven originally comes from a finance background, he's been spending the last several years heavily working on databases. We got a chance to talk to him in this KubeFM episode of what that process was like. What are the mental models that they used when it came to building the operator? After all, it's one of the things that, like I said, a lot of organizations are faced with this choice about how they're going to go about this, what's going to be the best fit. And like I said, the whole process is going to be involved with building that. This episode of KubeFM is brought to you by Learnk8s. Learnk8s is a training organization that offers hands-on courses to folks who are trying to level up their Kubernetes careers. Courses can be done online or in person and are 60% practical and 40% theoretical, given by some of the best instructors in the world. You have access to all the learning materials for the rest of your life, so you can enjoy them, revisit them if you decide it's a good time to freshen up. Now, let's get into the episode with Steven. All right. So, Steven, welcome to KubeFM. As we always start out with this question with our guests, what three tools would you install on a brand new Kubernetes cluster?

Steven: I would say Argo CD, Cert Manager, and then probably Grafana.

Bart: Okay. Regarding Argo CD, any particular reason why you wouldn't choose Flux?

Steven: Honestly, I haven't worked too much with Flux, and I'm more familiar with Argo CD. You know, I'm a big fan of GitOps and really like that workflow. I find it very easy, so I would probably install that first and then install the other two along with whatever else I need.

Bart: Fantastic. Good. Now, to understand better about what you do, tell us about where you work, the kind of role that you have, things of that nature.

Steven: Sure. So I'm currently a senior cloud engineer at QuestDB. QuestDB is a database startup. We have a time series database, and it's really meant for high-performance ingestion and querying. You can query the database using SQL, and we ingest also using SQL if you want, or in Fluxline protocol. We have a SaaS platform, which is mostly what I've been working with over the past year or so. And yeah, I mean, I highly recommend it. It's questdb.io and really enjoyed working there so far.

Bart: And how did you get into cloud native? You know, it's not something that everybody just stumbles upon. You know, what was it that brought you to this world?

Steven: Sure. I kind of have a non-traditional background. I started off actually with a finance and economics major in school. I went on Wall Street and really got my job because I knew how to use Excel VBA poorly. Eventually, I did a lot of analysis and moved on to a trading desk where kind of Excel VBA wasn't really quite enough for what I was trying to do. I had a mentor there. He said, you know, maybe instead of storing your data in an Excel spreadsheet, why don't you try this Access database? And after like two months, that wasn't enough either. So he's like, maybe look at, you know, SQL Server and C#. And it turned out I was programming way more than I was on the phone. So during a reorg, they moved me into IT, where I developed trading applications for bonds and analysis. Eventually moved on to hedge funds, realized, you know, this is what I want to do with my career. So I moved into the startup world. where I worked with two AI-related startups, I guess kind of before it was cool. And really fell in love with, you know, I did a lot of, a little bit of everything, front-end, back-end, DevOps infrastructure, and really enjoyed the kind of automation and DevOps kind of practices and decided to focus on that. So I've been doing that really full-time for the past three or four years. And obviously when you're working with DevOps, you know, it's... not mostly, it's mostly most likely going to be Kubernetes. And that's what I've been working with now for the past several years.

Bart: Very good. And how do you, you know, as someone also coming in with a non-traditional background, we can say, in terms of staying up to date with all the things that are going on the Cognitive ecosystem, learning these technologies, like the ones that you mentioned previously, whether it's Grafana or, you know, Argo CD, CertManager, what's your strategy? How do you do that?

Steven: It's a good question. I have a few main strategies. The first is just kind of like digging into it and more so on the coding side of just building like what they call a skeleton where I'm just like, let's build a POC as quickly as we can. And I kind of learn how things. hooked together from that POC. Reading the docs I've found over the years has been incredibly helpful. And then also, you know, YouTube videos, there are a lot of resources out there now for just people to explain at a more higher level, kind of how things work together. So it's definitely a mix of things, but there's so many resources out there that in so many different ways that people learn, you know, I don't know if it'll necessarily work for you, but I've found that to you know, be what has worked for me.

Bart: If you could go back in time and give one piece of advice, career advice to your younger self, what would it be?

Steven: Probably get a CS major. You know, the finance thing was great, but you know, my true love is computers and that probably would have saved a couple of years of kind of. you know, self-actualization, but no, I'd probably get a CS degree. I mean, to say that's, I don't think that's necessary at all. Probably just would have sped things up a little

Bart: Now, you mentioned that you work at QuestDB, right? So there's been a longstanding debate around databases on Kubernetes. This is KubeFM. I also had a previous life and spent a lot of time focusing on running stateful workloads on Kubernetes, which is actually how I found out about QuestDB. But some people out there still say running databases on Kubernetes is a bad idea. What do you say to that?

Steven: I actually wrote an article probably last year about why it's not a bad idea. Again, it's not for everyone. Persistent workloads on Kubernetes, since I've started working with Kubernetes, they have improved significantly. The CSI container storage interface really helps you to use a variety of different storage interfaces. Sorry, my dog is scratching at the door. Hold on. That's okay.

Bart: That's okay. More than welcome. Hey, what's up Nacho?

Steven: He likes being part of everything. That's good. I hope he might scratch to get out again, but I'll hold on.

Bart: That's okay. This is all part of the charm. So that's fine. You were saying about how things have improved over time with CSI becoming friendlier, let's say, for running persistent, safe workholes on Kubernetes.

Steven: Yeah. So you really have more flexibility now with the CSI and what you can do. It's still not. perfect in terms of, you know, mounting volumes and unmounting volumes, but kind of like taking a few steps back on that. People have developed a lot of operators for a lot of your most common databases, you know, I'm thinking really of Postgres, and I know that there are others to automate a lot of the like really complex things that you can do. So in terms of like running it on something, let's say like a managed service like RDS, versus using the operator. there's always a little more danger when you're managing your own stateful data. But you also have explicit control in terms of the level of snapshots you can take, how many you can take, and also all those different components and bits and pieces you have control of. So I would say today, it's really not something to be afraid of. I think if you haven't done it before, I wouldn't rush into it immediately and really take your time. read through all the docs and really understand what's happening behind the scenes, especially with persistent storage and the CSI and the inner workings of how volumes are managed. But if you can get comfortable with that, I think it's really worth a shot, especially when you're thinking about cost savings. It can end up being significantly cheaper doing that than using a managed service.

Bart: That in mind, like you said, there are a fair amount of resources out there, you know, people talking about this in the ecosystem. But if we're talking about provisioning and maintaining several databases on Kubernetes, how does one do that?

Steven: Well, that's kind of what I'm doing now and what we're doing at QuestDB with our SaaS offering. um so basically customers can sign up and create a database we actually have a free trial that they're able to do that and um that database needs to be spun up somehow and then that's what we're doing so My recommendation would be to use some sort of an operator or some sort of a higher level construct that lets you manage a database as a unit. Kubernetes is great in that you can really customize every tiny little bit of your deployment or StatefulSet or what you're trying to automate. But when you're dealing with that at such a large level, when you're dealing with multiple tens, hundreds of even databases, hundreds of databases, it can get to be really overwhelming. And that's really when abstraction starts to kick in and you're able to manage databases as a singular unit instead of the individual components.

Bart: We're thinking about things like rollbacks, the complexity that may lead to deciding what operator you might want to be using. Were there any issues with that setup? Does it work? If not, where might there be some problems? Could you comment on that?

Steven: Sure. So we wrote our own operator. And I guess let me take a step back. The article that we're discussing is really talking about how we moved from a provisioner, what we call provisioner, to an operator model. So generally, when a customer creates a database or modifies or deletes a database, a message is sent to a Kafka queue, and we have a provisioner that's reading from that queue. Previously, what the provisioner would do is it would break down that action into discrete steps. So, for example, if we're creating a database, it would create a StatefulSet. It would create all the networking. It would provision the node. It would provision the volume and kind of do all those pieces one by one. And when everything was complete, it would report back to the application and say that everything is good. Taking another step back, what makes us different about maybe your typical Kubernetes user is that we are single tenant. So we do have one node per customer database. We chose to be a single tenant, mostly to avoid the noisy neighbor problem. QuestDB uses things like mmap and other and io_uring and a lot of low level Linux constructs that We're not really, you know, we recommend that people only run the database in a single tenant fashion normally. And for our cloud offering, we wanted to do that in the same way. It makes the job of our provisioner and eventually operator a little different than what you would normally think of in Kubernetes because we actually care about our nodes. It's not necessarily like a herd of cattle. We have individual nodes that we do care about. And so. That is why we kind of have the provisioner in the way that we do and why eventually we had to move over to an operator model because of the complexities that were involved in managing and adding new features to the provisioner.

Bart: And could you also explain just the different, keeping a basic idea of what an operator is, how that's different from a controller?

Steven: Sure. So I like to really think of an operator as not just one thing, but multiple pieces that work together to accomplish a specific goal. So I'll start off with a CRD or a custom resource definition. And that is basically a custom schema that you can create and install that in your cluster. And that could be anything you want. So in our case, for example, we have a CRD that represents a QuestDB database instance. And that covers everything, as I mentioned, from the StatefulSet, volume, node, all of these pieces. Now you can create a custom resource based off of that definition, but the system needs to actually know what to do with that custom resource. And that's where the controller comes in. So the controller is listening to the API server for any changes in any of those custom resources that you create. So if I create or modify a QuestDB custom resource, the controller is hooked into the API server. and we'll pull the information from the API server about that resource and then perform custom reconciliation logic to make sure that the cluster state matches the desired state as specified by the custom resource. So for a more concrete example, let's say you want to run a 100 gig database on an M5 XL instance in EC2. So you can basically create fields on your custom resource definition, on your custom resource, that specify the instance type and the database size. When you then, kubectl, apply that custom resource into the cluster, the controller will pick that up and see, okay, is this instance actually running the correct instance type? If not, then we need to spin up the actual correct instance type. And is this database the same size? If not, we need to modify the volume.

Bart: One thing I wanted to clarify really quickly with CRDs, one person once upon a time was very active in the database ecosystem and also has an operator for a particular database on Kubernetes. One time he said that CRDs were his favorite feature on Kubernetes. Would you agree with that?

Steven: Yeah, I think I do. I personally really enjoy being able to harness the power of the Kubernetes APIs, controllers, distributed database, and do your own thing with it. That's a really powerful construct. And it really resonates with me personally, because I find that the work of writing these operators and controllers is really like the culmination of what I've been doing in my career. You know, I've spent a lot of time doing software development. I spent a lot of time doing infrastructure. I've managed my own infrastructure on .NET, you know, on AWS for a while, using Docker. And this is just a really nice mix of coding and distributed systems. And yeah, I guess it probably is my favorite feature of Kubernetes.

Bart: With that in mind too, you know, just for, you know, some people aren't familiar with this world in particular. So by using, you know, CRDs, that then ends up abstracting away complexities for the end user, correct?

Steven: Exactly. And that's one of the primary benefits of the new operator that we wrote, is that now instead of having the provisioner to manage each individual component, it just manages one resource.

Bart: And in terms of writing the operator, what resources did you use? Did you go out and evaluate other operators to benchmark, to see how they were working under the hood? What was your approach?

Steven: Kind of all of the above. I started off with the Kubebuilder book and also the operator framework documentation. That was really more of like the nitty gritty of how to do it. I also read a bunch of operator code that I found. And also I just read Kubernetes source code as well, the actual controller code and how things worked under the hood there. And then honestly, a lot of trial and error. I probably had at least two major refactors of this thing before we deployed it. And were they worth it? Like, absolutely. It really, there are not really a ton of best practices out there. You know, I found a few articles of people kind of writing a few suggestions, but I don't know. All in all, there's not a ton out there. And that's something that I'm actually doing personally on my blog is I've started kind of an operator 101 series to provide more of a higher level understanding of operators. So that people, when they do go to kubebuilder and operator framework, they kind of have more context around what they're doing.

Bart: Makes sense. But I think the, the point that you mentioned about a lack of standardization is kind of what makes it a wild west. And so you mentioned refactoring twice. I imagine there are other folks out there that for, with whom that would resonate. Were there any errors that particularly stand out that you would like to share with others so that they can avoid them if they decide they want to go that route of writing an operator?

Steven: Yeah, absolutely. Um, Keep it simple. That's really the biggest takeaway is the controller logic should really not be much more than what is my current cluster state? What is my desired state? what's the difference and fix the difference. If you're starting to deal with like state machines or different, you know, stateful patterns or you're dealing with a lot of different potential statuses on your objects. So like, you know, updating different pieces of status, it can start to get messy. very quickly. Another thing I would say is don't be afraid to create multiple CRDs. We actually have three CRDs, one for a QuestDB and then one for a node and one for a volume. And actually the QuestDB is the parent of those two. So by abstracting away a lot of this node-specific logic into a node controller and the volume-specific logic into a volume controller at a high level, all we really need to do is say Is the node there and is it the correct spec? And the same with the volume.

Bart: And in terms of, you know, once you did deploy it, what was the feedback? You know, how was that experience?

Steven: The feedback was no one noticed a difference, which is what I wanted. I designed the operator to basically create a carbon copy of what the provisioner was creating. So that when we were ready, when customers created new databases, we would create a QuestDB custom resource. Eventually, when we had kind of these older legacy databases and we were comfortable enough with the operator, all we did is just wrote a little script to create the QuestDB custom resource on top of that. And everything just kind of got assimilated into the managed resource.

Bart: So the migration from the old provisioner to the, to the, um, from the provisioner to the operator was not bumpy per se.

Steven: Uh, not bumpy. All right.

Bart: So in terms of your learning experience, you know, throughout this process, is there anything that you would have done differently? You know, if I had known whatever, it would have saved me a ton of time. Is there anything, you know, hindsight you're like, you know, if I had to do it all over again, I'd do that differently.

Steven: Yeah, I think I probably would have. read more documentation and not just jumped right into it. As I mentioned in the beginning, I have a tendency to kind of just jump right into the code and build a POC. And that was, you know, it worked, but that's why I refactored it twice. I feel like if I had just taken a little more time to read some of the more, read some source code from the start, read just some more operator. you know, information and try to get a really higher level understanding of what was happening instead of just jumping right immediately into the details probably would have saved me a little bit of development time.

Bart: And I imagine that would be your advice to someone if they were going to write their own operator or would there be any additional advice that you would share?

Steven: Um, well, I mean, You could probably, I think a great place to be, honestly, would be my blog. It's a little self-serving, I guess. But I'm writing these articles for people who want to build operators and don't really understand or don't have really a background in operators or know what they are.

Bart: What's next for you?

Steven: I'm finishing up parental leave. And then... Well, I have to think about that. Okay.

Bart: But I guess, okay. So that is very important. So, you know, life plan wise, parental leave is coming to an end. Do you plan on writing more blogs, building more operators? What do you think from, you know, in terms of Kubernetes, what are you going to be doing next?

Steven: Yeah. So I'd love to. I'm planning on continuing my Operator 101 series on my blog. And then also I've been spending a lot of time with the CSI internals and working on building on integrate on working on getting a better understanding of what's happening with the storage itself.

Bart: That's good. That's good. Storage is not something that people often, that you'll see storage and easy in the same sentence. So that's never something you can know too much about. Cool. And if people want to get in touch with you, what's the best way to do it?

Steven: So I have a Twitter. It's my last name, S-K-L-A-R-S-A, Sklarsa. And then I also have a blog on sklar.rocks.

Bart: Perfect. Well, Steven, thank you very much for your time today, for sharing your knowledge, your wisdom, the difficulties that you went through, and all the while making it easier for other people that are facing similar challenges. Thank you for joining us today on KubeFM.

Steven: Thank you so much.