Clusters are cattle until you deploy ingress

Bart: What should you do if you need an ingress, but you also want to keep your infrastructure reproducible? On top of that, what's the whole buzz about GitOps? Why should people use it? What are the benefits? What about the fact of talking about application lifecycle, if we're talking about environments, talking about all sorts of things that someone's going to have answers for? And that person happens to be Dan Garfield. Not afraid of hot takes, also some interesting thoughts about crypto, fairy tales, and how to make your own bacon. He's an Argo CD maintainer and rocking all things open source at Codefresh. This episode is sponsored by Learnk8s. Learnk8s is a training organization that offers both in-person and online trainings for folks who are trying to level up with their Kubernetes skills. The courses are 60% practical and 40% theoretical, and all the students will have access to materials after the courses for the rest of their lives. Check out Learnk8s.io for more information. Dan, welcome to KubeFM. If you have a brand new Kubernetes cluster, which three tools would you install?

Dan: oh boy first three tools i would install would be for sure argo cd uh if it's on aws then i'm probably gonna install Karpenter next just to keep my cost down um and then uh i suppose if it's an on-prem then probably longhorn is the is the next one though i guess i got it i got to do ingress too so but but yeah i definitely argo cd and then depending on what i'm doing one of those other two

Bart: It's interesting because, you know, in the podcast that we recorded, almost everybody when asked that question, whether it's Argo or Flux, something GitOps related is mentioned as someone who's well known in the space and is very active in the space. Why do you think that is that people say they just can't do without it?

Dan: uh well it's so so for me um the whole workflow for how i deploy to kubernetes and how i recommend people deploy to kubernetes and manage their software is through argo cd and when i start building a cluster um i think we you know some people might have the habit of like oh i'm gonna go in and start kubectl applying and it's like okay like you could do that um you know if you're bootstrapping the cluster with like terraform then you might have your you know, use the Helm provider to install a bunch of Helm charts and stuff like that. But, um, So with Argo CD, I have really granular control over how things are deployed to my clusters. So the most common bootstrap pattern that I see, even from Terraform users, is Terraform bootstrap cluster, Helm provider, bootstrap Argo CD with some preset repos that it's pointed at. And then it just automatically brings up everything else. My cluster, over my shoulder, for those that are listening, they won't be able to see it, but I've got a... my Kubernetes cluster is visible on the screen. It's back here, my home lab. And this is running Argo CD. And I actually use Argo CD autopilot, which basically sets up your repository for you. And so if it's like very little work to manage, because if it got nuked, and I actually had it get nuked last year, and I ran Argo CD autopilot, you know, bootstrap, you recover, and then it was back. So... it's just nice to work that way. And then if you're debugging something, having the ability to like turn on and off sync, really quickly reset the application that you're working on, view the logs, all that stuff through the UI is certainly really nice. So yeah, I think it's the most important tool for Kubernetes for me. I'm biased. I'm an Argo maintainer. What else am I going to say?

Bart: Of course, of course, of course. But like I said, it just... you know, we've done a fair amount of these podcasts and with people that are quite experienced, it seems like GitOps is just industry standard at this point that you can't do without it. And, and, and not only that, but I would have to listen to the other podcasts to see how many people mentioned as the very first tool, you know, like they might mention others, you know, cert manager, um, some might say, you know, Kyverno or OPA depending on their preference. But like I said, the GitOps factor, I would say out of 90% of the people we've spoken to so far, it's come up. Um, Now, taking a little bit of a step back, for people who don't know you, can you tell us more about who you are, what you do, and where you work?

Dan: Yeah, so my name is Dan Garfield. I am the co-founder and chief open source officer of a company called CodeFresh. We're Argo maintainers, and I myself am co-creators of the GitOps standard, the GitOps working group, OpenGitOps. And we launched the company about seven years ago. with the goal of helping people deliver software more efficiently, more effectively in cloud-native tooling. And so that specifically has been Kubernetes. And as an Argo maintainer, I work a lot on SIG security. I helped start SIG security. And then I also helped run ArgoCon and those community initiatives. And... Yeah, what else? I don't know. I live in Salt Lake City and I like to snowboard a lot. And I've got four kids and waiting on number five.

Bart: That's a significant project. And obviously we'll be looking forward to connecting with you when we're celebrating KubeCon in Salt Lake City later this year. Before starting your company, how did you get into this world? How did you get into Cloud Native?

Dan: Oh, gosh. So I've always been a programmer since I was a kid. And I found that I was kind of a mediocre programmer. And so I tended to work more on the business side. And over there, it felt like a superpower to be able to do that. So through various whatever, I found myself running enterprise marketing at Atlassian. And when I was there, we were going through and releasing Data Center, which was the clustered version of Atlassian's tools. nobody really cared about it internally in the beginning. And we were able to get it out there and it suddenly became the most profitable, most effective revenue driver for the company. So suddenly people cared a lot. So that was really cool to be part of that experience and to be there going, taking the company public and all that stuff was great fun. And I don't... overblow my contribution on any of this. You know, I wasn't, I was only there for a little less than two years. So most of this was the work of other people, but at any rate coming out of that, I was looking for the next challenge and I knew there was something really interesting happening with containerization and it was kind of a bellwether for like, okay, there's a technological shift happening. and it's going to be a big earthquake. I don't know what's going to be relevant. I had some friends starting companies around containers. I was talking to them. And Raziel, who's the... I always say I'm the co-founder, but Raziel's the founder of Codefresh. We're not both co-founders. He's the founder. I'm the co-founder, right? But he reached out, and we got in touch and started talking to each other. And he shared his vision for how he thought... you could encapsulate the software journey around containers and what you could do with that. And I thought that was just super cool and wanted to be a part of it. So joined and we launched the company and Codefresh early on was focused a lot on the CI part of the equation because for most people to do CD. they have to have decent CI. And most people at the time didn't have decent CI. I'm not sure that most people now have decent CI. But so that was kind of the first challenge to solve. And we shipped a really exciting CI product, especially for the time. And then we started working on the CD components and how we could expose what was happening in Kubernetes. And you have to understand when we started this company, Kubernetes wasn't the winner. yet. It was like, oh, Rancher could be the winner because they had their own thing that was not Kubernetes. Or maybe it's going to be OpenShift. They had their own thing that wasn't Kubernetes. And maybe it'll be Docker Swarm. And maybe Mesosphere is going to take over the world. And we looked at all the fundamentals of the technology and we felt like Kubernetes really was the strongest. So we made a very big bet on Kubernetes. And over the following year, after we made that decision, slowly we saw Rancher was like, you know, we're actually going to be rejiggering this on Kubernetes. And then OpenShift did the same thing. And then Amazon was like, we're going to do EKS. And so all of that. that crystal balling from Raziel really turned out. And yeah, the rest is history. We got into GitOps and Argo CD and pushing that project forward and helping it join the CNCF and helping it graduate and all that stuff.

Bart: And obviously this is all within a relatively short span of time. Seven years may seem like a long time and also building a family at the same time. There are a lot of things happening simultaneously. in an industry that moves so quickly in a world that's so fast paced as a maintainer, as a, you know, as a co-founder, how do you stay on top of all the things that are happening constantly in the cloud native ecosystem? Is it through blogs and through podcasts? Do you read, do you watch, what do you do?

Dan: Yeah, certainly, you know, blogs, Twitter, um, are, uh, you know, podcasts are all really, really great. Uh, I find that I learn the most by talking to people one-on-one. So I'll do, I've done this a couple of times where I've put out to the community, like, hey, if you need help doing your Argo implementation, you know, reach out. I'm happy to chat. This isn't, we don't sell services really. So this isn't a sales call, but like just happy to look at your architecture and share my thoughts on it. And that's given me the chance to talk to. just tons of people. And so I normally interact at least a little bit with Codefresh users, right? We have a customer success team that's primarily, you know, on that, but they'll bring me into phone calls and stuff. So I'll interact and help with people adopting this stuff and get a lot of insight from them. But then talking to other people in the community and hearing their needs, that's really important. And then, unfortunately, the most effective way that I've found to learn is to put out wrong opinions or wrong things, and people will show up. So I did this. I put this into practice the other day because I was working on a project with AI. And I was like, man, is every Docker image just with any model going to be giant? Because you have to have... a pytorch installed and it's just a big thing. And so I put out, put it out there. I was like, ah, this is a pain in the butt. Cause everything has to be so big. And suddenly I had like 15 people like, oh, well here's ways you can make a smaller. And I was like, ha ha. You know? So if you're willing to, to put yourself out there and be wrong, you can learn a lot faster.

Bart: I think that's a really good point, though, is that, you know, being vulnerable and not worrying, you know, that if someone maybe responds in a strange way, that's their problem, it's not yours. And you do others a favor by putting those questions out there because somebody else probably has that doubt, I would imagine. That's a good practice. We'll probably dig into that a little bit more later on. But one other thing I wanted to ask is that if you could go back and give your, you know, younger self seven, as many years ago as you'd like to, you'd like to give yourself one piece of career advice, what would it be?

Dan: gosh what career advice would i give myself probably learn faster um just just be more willing to be wrong and have less ego associated with your ideas um you know fight for the stuff that's important but i think that as you build a business you always run into stuff that doesn't work and then you'll be like oh well what does work and you have to look sometimes you have to look far afield to figure that out and going through that process, it's like, it's like being a sharp stone rolling down the mountain and you're going to come into collision and it's going to knock the sharp edges off. And that's how you end up getting smoothed out. And, uh, so I don't know, I have some patience for that. And my other piece of advice would be, um, probably, uh, the, the most important decisions you make in a business are who you hire because the wrong hire, it costs a lot more than their salary. And, um, certainly, you know, had some pitfalls with that, but we've been, we've been pretty lucky to, we, you know, we work with like the best people in the world. I love, I love people I work with. And, um, I always tell people that when it comes to a job, uh, I always, I think there's three considerations. The number one in my mind is the people. If you're standing in the right room with the right people, you're not going to be able to do the job. You're not going to be able to do the job. everything else is going to fall into place. And so I think that's the most important. Number two is mission. Okay. Are you going to work on something relevant? But if you're standing with the right people in the right room, probably going to be working on an important mission that you feel good about. And then number three is compensation. You know, that's, you got to feed your family and stuff. But I think, I think if you make the decision in that order, you're much happier in life.

Bart: I think it's sound advice and, and definitely, you know, the part about hiring is something that for a lot of business owners is. you only really learn by going through some difficult decisions and difficult situations and understanding how to optimize for those in the future. And the part about the right people in the right room couldn't agree more. And everything gets easier when you're surrounded, in my case, surrounded by people that are smarter than me or have other experiences and bring things to the table that I don't. So yeah, really, really like that. It's good. Now, our topic for today, though, you wrote a hot take on Twitter. That said, Kubernetes will be treated as disposable until you deploy Ingress, then it becomes a pet. So can you tell us what audience was that tweet directed towards?

Dan: The void that I shout into? No, I think it's just I was thinking about it for myself because... sort of the root axiom of engineering is that the problem is always DNS, right? And so the inverse of that is essentially like your cluster becomes real once you set up the ingress, which kind of implies that you set up DNS. and once you've done that then it's special because you no longer have it just part of like like if you don't if you don't have anything pointing at it you know what i mean um then it's like, okay, this is like a dev cluster or something where maybe I'm just working on something just for me. And so I don't care if I lose it. But as soon as you set up Ingress, you're basically running services on it. And once you're running services on it, then you have to care about it in a much more substantial way.

Bart: for some folks out there that may not be familiar with the model of, you know, cattle versus pets, can you explain a little bit more in detail about what, um, pets might have to do with Kubernetes and ingress?

Dan: Yeah. So the cattle versus pets analogy, which I think is universally detested by like vegans and, um, which I totally understand, you know, uh, but, uh, but, but the analogy is essentially that cattle are the only animals that can be used to feed the cattle. So, um, and so, and so, and so, if you think about what cattle actually are, like you don't necessarily, as a person consuming a hamburger, you don't necessarily care about the name of the cow, right? Because all the cows are essentially indistinguishable from each other insofar as their hamburger.

Bart: We're getting a lot of hate from vegans.

Dan: Yeah, I understand why they don't like this analogy. I didn't come up with this analogy. I'm just explaining what the analogy is, okay? um this will be the most popular podcast we were recording yeah but it's it's uh they're essentially commoditized and so you don't necessarily have to have a relationship with it but a pet has a name you're like oh i care a lot about what happens to my pet and um i want to make sure that my pet's okay and all this kind of stuff so the way that you care about it is different a pet is not replaceable you know when you're a five-year-old if your pet goldfish dies you Maybe your parents will replace it without you noticing. but if it's a dog, you're going to go through a heartache of replacing that dog. When it comes to software, the analogy is essentially stateful services generally. It's like, what are things that are disposable versus what are things that you have to care about? I mentioned that my Kubernetes cluster, I'm running Argo CD and everything bootstrapped. If stuff breaks, I can rebootstrap it. These nodes are all... cattle in this analogy, right? I don't care about any of these nodes in any specific way. If one of them broke, it's like, okay, I'll just unplug it and plug in a new one because they're not running anything special. There's nothing bespoke about these nodes. My cluster, on the other hand, I care about its state. and I care about the configuration that creates that state and how I get it on there and how I keep it up and running. So in that way, while every node in my cluster is essentially disposable, my cluster is not disposable. I actually, you know, once I've deployed Ingress to it, I start like having thoughts about keeping the uptime and making sure that, like, I care about the metrics all of a sudden associated with the cluster. So that's, I guess that's what I mean. Maybe that's too long-winded. Does that make sense as far as cattle versus pets and why it applies?

Bart: I think, like you said, yeah, it's essentially without any, you know, it's just a matter of, you know, dietary preferences being involved in the understanding of it is it's like you said, the, the feeling of closeness or that there's a higher degree of, of, I don't want to say importance, but, but yeah, of proximity and that there's an element of care that's involved. So now, now can we bring it though? What does Kubernetes and Ingress have to do with all that?

Dan: So the ingress is, it's an interesting resource within Kubernetes because it's very often shared. If you think about Argo CD, you have this delegation of what you're deploying into applications. And every application can have its own rules for how it's deployed. So you can say like, oh, this application only will be updated when I click sync. And this other application will automatically be updated. whenever it needs to be updated. And I can have different rules for what kinds of differences I'll allow. So for this application, I've got a pod autoscaler running. And so I don't care if I change the number of replicas in Git, I want the pod autoscaler to handle it. So you can do all this stuff, right? With an ingress, you have a single spec that describes what you're doing. how traffic comes into your cluster and how it's routed to different services. And so that in some ways breaks the application delegation model. because in order to route to this application, I have to update my ingress. But my ingress needs to be updated for many different applications. And so part of the reason that once ingress is deployed, or at least that I think that once ingress is deployed, that you start to care about the cluster is because you're now having to... manage juggling the needs of several applications by setting up their ingress in this central way. And there actually is a feature that is in discussion right now for Argo CD that is delegated server-side. field permissions, which basically what it boils down to is you could have an application that can modify components of another application. So you could say this application can add a route in an ingress, even though it's not part of that application, and you can delegate that to these other apps. I don't know how I feel about it yet. It's still a discussion, and we'll see if we... move forward with that if there's another way to solve it. There are other tools that do this. By the way, like competing to Ingress, you've got, I think Contour is an example where every route is its own custom resource. And so each application could just own its own route. And so if you're using a tool like that, then you don't have to have that delegated permission thing. But yeah, once... Once you set up Ingress, which is always kind of bespoke in a way, because you have to have a relationship between the centralized resource and all these applications. And on the external side, it means that you're thinking about DNS. Once you've got DNS going, now it's bespoke. Now it's special. Now, because even if the DNS is essentially operating off a wildcard. you need it to route there. And so it now needs to point at something. You know what I mean? So once that happens, then the cluster is bespoke in some way. It doesn't feel as disposable.

Bart: And is there any recommendations if folks want to keep their infrastructure reproducible that are going to be needing Ingress and still want to keep it reproducible? Is there anything that you would recommend there?

Dan: Well, I certainly like, you know, you can abstract and use wildcards. And technically, like, I guess you can deploy an Ingress and nothing external points at it. But I do like the idea of having these things self updating. So to the extent that you can use a tool like crossplane, or Google Cloud has their own tool that's called Config Connector or something. Well, I can't remember the name of it, but basically it lets you represent non-Kubernetes resources as Kubernetes objects. So I don't think it's a bad idea to have that be bootstrapped as part of your cluster. and then just make sure that it's, it's provisioned in such a way that, um, it'll dynamically create the stuff that you need. But, you know, at some point. you've got to update your name servers. You know, you got to point stuff. So the cluster may be reproducible, but the external components may not be. So I don't know. It's just a tricky part of the whole operation that isn't perfect in my mind. And maybe, and you know what? I'm putting it out there, Bart, because someone's, we're going to have a bunch of people say, actually, there's a great way to do this. And they're going to come back with all the great ways to do it. But in my experience, it just hasn't clicked for me yet.

Bart: But at the same time, where would GitOps and perhaps Argo CD come in to potentially alleviate this problem?

Dan: Well, if you can represent resources as Kubernetes objects, then Argo CD can reconcile them. And so this is why the crossplane plus Argo CD, or if you're not using crossplane, let's say vCluster, if you're doing multiple clusters, or cluster API, if you're provisioning additional clusters, or there's a bunch of different infrastructure management tools that can connect into Kubernetes. what you're doing is you're creating a new data store for your infrastructure state and you're storing it all in etcd, right? So you've got your cluster, you've created your objects and their state is stored in etcd. so these things like crossplane will rely on these controllers that will go and make sure that those resources are created outside of the Kubernetes cluster. So if you do that, well, Argo CD is going to be the thing that syncs those resources. And that's what you're going to use to set up your policy for how those resources are updated. And if somebody goes and modifies them, they're going to be, you know, if you have auto auto sync turned on and they'll be self-repairing. And so they're a lot more resilient. The issue with like Terraform that a lot of us have had is Terraform state gets corrupted relatively easily because it's not really monitoring constantly. Crossplane, a lot of Crossplane uses Terraform under the hood, right? So it's not that the primitive of Terraform is wrong, but the data store is really important and how you keep it updated is really important. And Crossplane has managed to do it in such a way that the data store is not getting corrupted. And so you're always looking at this. reality of what truth is. And if something changes, it is reflected and shows up as now being out of sync from Argo CD. And now you can have your policies set up for how that's going to be reconciled, how it should be updated. And that's going to inform the controller about what it should do next. So it's a big part of the equation for managing this stuff. If you want to use that CD as your data store, which is a great pattern, honestly, I think it is the future. That's what it feels like to me.

Bart: And, but let's say if the issue doesn't stop with the ingress, if you start managing state secrets, certificates, etc., you may eventually end up having a pet cluster. If that weren't the case, tools such as Velero would probably be much simpler. Would you agree with that?

Dan: I'm not familiar with Velero. Tell me about it.

Bart: Ah, if you're not familiar with Velero, in terms of if you are into the world of backup restore, anything of that nature, since you mentioned it, CD also mentioned custom resources earlier. Is that something you've played around with at all?

Dan: Oh, so it's for data migration, data protection. So would I use this with Longhorn? So for example, on my home lab, I've got Longhorn deployed and it is backing up volumes to an external data store that's essentially S3 bucket. It's a Minio store, but because it's on site, but so I've got it backing up there. And so I could restore from there with Longhorn. But would I use this with Velero to do that automatically? Or is it a separate tool?

Bart: Yeah, so Velero is a separate tool. I'm not being a total expert on this, but I'm just curious, like, for example, because you mentioned your home lab, right? Having the problem that, you know, it got nuked. How did you go, you know, disaster recovery in terms of backup, you know, persistent volumes, things of that nature? This is where this is going to come in handy.

Dan: Yeah, so in that case, I've used Longhorn to restore volumes when needed and just set up a policy around that. But I just pulled it up. I'm looking at it. It looks like a cool tool for doing data migration, which is. I mean, that's the other thing is once you have data on it, then it's, that's, that's the other time when it starts to become more of a pet, right? Because you have to take care of that data. You have to migrate and stuff. And a lot of people just don't. do it they just they're just like if it's stateful i'm just not going to do it in kubernetes um and luckily a lot of the stateful stuff in kubernetes that i do is stateful ish where it's like if it's lost it's not necessarily a problem it has to maintain state while it's running but it can rebuild its database you know um so it may not necessarily be a problem if it gets lost and most of the stuff that i do is like that i have a couple of Well, now that I'm thinking about it, almost everything is recreatable that I keep track of. And anything that needs to be persisted between stores is kept in Git or maybe a secret store that is versioned and immutable too.

Bart: just interesting to keep in mind, particularly since you had that incident with your home lab. And then on top of it, you know, given what we mentioned earlier about pets versus cattle, I think one way of thinking about, you know, stateful and persistent volumes, maybe think about it, pet pictures. You know, you've got your pet pictures that are being stored on somewhere. If something goes wrong, you know, once again, there's that more, you know, endearing sort of connection. But anyway, that's all good. If we keep moving in, if we thinking about, you know, clusters as cattle, if should someone be obsessed with treating their clusters as cattle if they're a small startup or would be click ops would click ops be enough to save the day so i think it's going to depend on the use case um so for example

Dan: if you're using, one of the projects that I love is vCluster, which allows you to deploy a virtualized cluster on an existing Kubernetes cluster, which by the way, relating back to ingress, ends up saving you a ton of money on ingress because you can just have one ingress for all of that stuff. And suddenly you save a ton of cash if you're using something like AWS. But in that scenario, you're going to have to go. One of the use cases for that is development clusters. So I may have a whole bunch of Vclusters that are being automatically generated for developers to use. And in those cases... I can architect them in such a way that I don't, that they are disposable, that they can come up and come down. Or another example is like, let's say you're using an application set in Argo CD. An application set allows you to generate applications programmatically. And so what I can do is I can say, for example, for every pull request open on my Git repo, generate. a set of applications, and that can include a virtual cluster with a full stack and then post that back onto the pull request. So I get a full pull request environment in its own cluster. In those cases, those clusters are... are unique because they contain the information of the pull request, but they're completely disposable and recreatable. So in that case, they're very much like cattle and I don't really have to care about it. They can just, they can spin up and spin down. And there are scenarios that you run into ingress is one where it can create problems for those. Once, if you need to deploy ingress and pass that onto these Vclusters, suddenly you have some bespoke configuration that's going on. that you have to track out kind of outside that process. But, um, but yeah, overall, I think having, certainly having the hardware be disposable, right? Like I mentioned in my case, I have one, I have one Kubernetes cluster on site, right? So for me, that cluster is a pet, right? I will, I need to take care of that cluster, but all the components of it are cattle. The nodes are cattle. I can dump them, throw them away, reconfigure them. It'll keep running. And that's. that abstraction is part of the joy of Kubernetes, kind of part of the point of Kubernetes. So if you can get to abstracting away the clusters so that they're not bespoke, so they're not special, so that they can be spun up, recreated, no problem, then you can start, even though they might be a bit pets because there is ingress deployed because DNS is pointing at them, because they're recreatable, because they are... commoditized, they can start being treated a lot more like cattle, which is nice. So the more you can do that, the more you can abstract it, that's an improvement. And again, I'm not making any commentary about eating meat.

Bart: None whatsoever. Impossible burgers are welcome. It's all good. Good. Now, how would you fix this if you could go back in time and change anything that you wanted? Let's imagine you could create a scenario where you wouldn't have to have written that tweet that you wrote.

Dan: So one of the things that we talked about, and I mentioned that we have this feature in Argo CD that's being discussed right now about delegated field permissions happening server-side. What we're solving there is essentially a problem you could look at as a problem with Kubernetes architecture. because ingress has been created in such a way that it doesn't allow for external delegation of its components, even though most people actually use it that way. So if I could change something in the world, I might have split ingress into an additional resource that would include routes as a separate definition that could be managed separately. And maybe we'll go open a pull request to do that in Kubernetes. I don't know. Maybe that's a better way to do it. I'm trying right now to look for other use cases when delegated field permissions would make sense. Ingress is the most obvious one. And that one essentially exposes a kind of a flaw. Or let's not say a flaw. Let's say an area for potential improvement for ingress, which is that if I could create separate routes and separate resources, that actually would solve this problem without me having to change anything in our CD. So that would be one way to do it. Yeah, I think that would be a good solution. And I think I mentioned that that's the way I think Contour does it, is they actually have those created as separate resources. So they have second mover advantage. They learned from Ingress. They made some improvements. Maybe we can bring some of that into Ingress, or maybe we should be using... maybe we should be using contour or some other tool like that where, and there, there are actually quite a few cool little service mesh ingress provider tools out there. So maybe that's a, another way to go.

Bart: And if you had to build a cluster from scratch today, how do you work around these issues whenever possible?

Dan: You either just bite the bullet and don't work around the issue and just say like, yeah, well, I mean, it's not necessarily a big deal if you're just deploying a single cluster to set up ingress and point your DNS. But if you need to do it 250,000 times, then you might want to think about re-architecting. So for another example, with Codefresh, the way that most people use us is what we call hybrid. Now you can deploy our full stack on-prem. Right. But most people use hybrid. And the way that works is they deploy our GitOps agent that's based on Argo CD onto their cluster, and then it connects to our control plane. And one of the things we actually offer is a hosted ingress. And so what we'll do is you have 5000 instances of Argo CD, you could deploy those to 5000 clusters, set up 5000 ingresses, and you're probably going to pay a bucket of cash just to do that. we have a bunch of customers who are saving money just because we have a hosted ingress where all of those Argo CD instances automatically just go through that ingress. So they don't have to deploy it. So my example there is instead of doing bespoke ingress, maybe there's some kind of tunneling system that would make more sense for your use case. That's going to depend a lot on what you're doing. But in the case of distributed large numbers of Argo CD instances... that hosted ingress is killer. That's a killer feature and saves people a ton of money and, and a lot of configuration headache. So yeah, re-architecting is, is always an option and architect for the stuff that makes sense for you.

Bart: Good. Well, congratulations on making it so far in the podcast. Now we want to just get to the closing question, but we're going to take a few different angles. So, you know, engineering tech news is often polarized between two extremes. For example, it's common to find an article that will cover, you know, the best practices for, you know, a certain technology or, you know, a certain area of the industry. And then a few years later, there'll be a follow-up piece that will be titled the thing that we talked about a few years ago. Well, it's actually quite harmful. Do you find that this is true? And how do you cope with, you know, the anxiousness of trying a new tool or best practice only to find out a few years later that perhaps it's wrong?

Dan: Okay. So I guess there's a sunk cost fallacy in operation there. I've seen certainly situations where people adopt a tool and it ends up in two years that they need to replace it with something else. We went through a conversation. I don't think I'll mention their name though. They did talk about this quite a bit publicly. There was a company that went through a big transition to using Argo workflows to orchestrate all of its CICD and all of its deployment. and after they went through that transition, they realized how Argo CD would actually be a better fit for most of that use case, and so they started transitioning again. In their case, their transition into Argo workflows served as the jumping-off point to get them to Argo CD. you know, I don't think it was a waste of time. They went through that and they re-architected a bunch of stuff and then it made their adoption of Argo CD easier. Could they have gone directly to Argo CD? Yeah, but from where they were, that wasn't clear. So sometimes you have to, sometimes I think it's better to run in the wrong direction together because you get to the answer quicker and then you can focus on going to the right answer. You know, you can't always see, the answer from where you are and you have to go through tools. And I just think that's part of the learning experience. And I wouldn't, I certainly wouldn't feel bad about it. people that make a mistake have to live with the mistake once, but people that have to live with regret around that mistake have to live with it a thousand times. So it's better just to take the experience and be like, okay, we learned from it and let's move on. The other thing is a lot of times you get value out of something in the meantime anyway. So even that, that transition Argo workflows was a huge improvement for them. So even could they, could it have been gone better and they could have maybe gotten ArgoCD earlier? Sure. but they did get like a whole year and a half of huge improvements to their software delivery in the meantime, because of the work they had done. So, um, you know, except the wins and don't look them in the mouth and just be like, you know what? We had some wins. We made some improvements. Let's go back and do another round of improvements and we'll just keep moving. And, uh, sometimes you have to migrate away from a tool. And, um, ideally it's those mistakes are only fatal if you don't recognize them. Right.

Bart: In terms of digging into things, we did some digging on our own and found that you have a blog that hasn't been updated in a while called Today Was Awesome. In 2015, you wrote a post about Bitcoin. Now that was quite some time ago. And at that point in time, I believe Bitcoin was priced around $450. Am I currently speaking to a crypto millionaire?

Dan: No, not quite. No, I wish. crypto is funny dude because it's it's almost religious for people if you talk about crypto they're like oh crypto crypto bad i hate crypto it's all scammers and nonsense and bs and it's like well there there are a lot of scammers and there's a lot of nonsense there's a lot of bs um So I don't blame people for being angry about it. No, I was always very cautious. I did a little bit with Bitcoin early on. I mined it, wanted to learn more about it and see what use cases existed. I had a really great experience being a mentor at Hack the North, which is the largest hackathon in the world. It's held in Vancouver, Waterloo, Canada every year. I don't know if it happened during COVID. I wasn't part of it. But just before COVID, I was at a hackathon there and there were tons of projects doing stuff with crypto and they were all building on Ethereum. So I thought that was really exciting. So I actually sold all my Bitcoin and bought all Ethereum, which at the time was a good maneuver and turned out to be a positive. I think I like doubled what I'd put in. But unfortunately, not a millionaire. Those Lambos are still inbound, still waiting for the moon.

Bart: The Lambos will have to wait. Now, also digging into this blog, you know, you have some other posts too. And so just to folks know, it's not exclusively about crypto or anything like that. But you wrote an article about what are we really supposed to learn from fairy tales? So your article, what are we really supposed to learn from fairy tales? Take a look at this right now here. We've got everything from Rapunzel, the Red Riding Hood, to the Frog Prince. how do you come up with this stuff? And this is, I imagine, uh, I'll check the date again. This is, well, this is 10 years ago. This is 2014. This is not chat GPT. If you, you know, type and stuff in looking for answers there. So what, what like spurred the moment of like, ah, definitely didn't get this in my blog.

Dan: I don't know it's so long ago um I don't know I I uh my wife and I are very humor we spent a lot we communicate through humor humor a lot and so we're always just making jokes to each other and so I don't know I thought I think I thought that it was funny that when we look at fairy tales um you know, like when you tell, when you teach your kids a moral lesson, it means something, but a lot of fairy tales are from like some bygone era of culture that isn't computing anymore. And so the lesson that they're teaching doesn't quite make sense to our modern ears. And, um, so it was just taking that idea of like, these things don't. they don't always make sense in the current cultural context. And so taking them the wrong way can make them seem funny. So it was just a joke, I suppose.

Bart: Cool. But I think it's nice to dig into those things and think about, you know, how can we connect them to modern times? And we see it again and again, specifically in the sense of fairy tales, whether it's through movies, through superheroes, things like that. We see these stories coming back and perhaps in a more modern adaptation. But I also just like the fact that you put this stuff out there. You know, I think it's, and it's encouraging going back to what we're talking about at the beginning. You know, it's okay if you put something out there, the reaction will be whatever it is, but you'll, you'll walk away with, uh, with something better from it, I think. Now what's next for you? Can we expect more fairy tales, more Lambos to the moon, more snowboarding? One thing that we didn't talk about actually yet, we didn't talk about a tweet that I saw that you put out recently about making your own bacon. Tell me about that.

Dan: Yeah. Yeah. So I, uh, I make my own bacon. Um, it's really easy. Actually you buy pork belly. and I just get mine at Costco. It's not, you know, I don't have to go to the farm or anything, but you get pork belly, you brine it in the fridge, you cure it for 10 days, seven to 10 days, and then you just smoke it for like two hours. Then you've got a bacon slab that's bigger than your chest, bigger than your body. And you can slice it any way you want. You can slice it thick cut in the French style, like those lard on cut bacon, which is so good on like ramen and stuff. But... the big motivator that started me to doing it was because a lot of bacon has nitrates in it and nitrates are linked to cancer. Nobody wants to hear about diet stuff, but anyway, it turned out that making bacon was actually super easy and I could just leave that stuff out and it tastes delicious and it's... far higher quality than anything I got from the store. And it probably costs less than what I get from the store. So it's kind of a win, win, win. And now it's like, um, I have my freezer is just full of bacon. And if I need to give a gift to anybody, I'm like, here's the bacon that I made. And it turns out that's a really nice gift. Uh, and people can cook it. And I always get nice reviews when they come back and have sliced it up. And they're like, this is like different. This is not just bacon from the store. This tastes really different, really good. This is special. So it's been a lot of fun. I like it.

Bart: That's great. And I think it's all, you know, dietary choices aside, you know, you are what you eat, thinking about where your food comes from and how you can be involved in the process, whether you grow your own food, whether you're making it in this way and turning those things into a gift so other people can enjoy it, have a different experience. I think that's really nice. Apart from that, what's next for you?

Dan: Okay, so this year for me is all focused around environment management and promotion. And people, we in the Kubernetes world, we've been thinking about applications, clusters, and instances of Argo CD to manage all of that. And... what we're working on right now is this kind of paradigm shift where instead of thinking about applications, thinking about products. And a product in our world is an application in every environment that it exists in. So if you deploy to dev, and then it moves to staging, and then it moves to production, you're deploying the same application with variances three times, right? that's what we call a product. And so instead of thinking about the application just where it lives, thinking about it in terms of its life cycle. And instead of thinking about clusters, thinking about an environment because an environment might be many clusters, right? If we have users who are deploying to retail shops, right? So you can think like, you know, Starbucks, Chick-fil-A, Pizza Hut, a lot of these places have Kubernetes clusters on site. And if I deploy to US West, Well, US West might be 1300 different clusters and 1300 different Argo CD instances. And I can abstract all that way by putting them all into the environments bucket. So the thing that we're really focused on right now is helping people scale up and build out their entire workflow using environments and setting up that relationship. And it's really... I mean, what we've been doing and showing to people, like people's faces melt off of their skulls. I mean, it's like mind blowing. So I'm really excited about that. We've got ArgoCon coming up next month. We're going to be showing off all this stuff in Paris. So going to Paris, showing off environments, doing some snowboarding, and then making it back in time for baby number five to be born. That's the plan.

Bart: That's a big plan. So 2024 is actually packed for you. If people want to get in touch with you, what's the best way to do it? Slack, LinkedIn, Twitter, what's the best way?

Dan: Twitter is probably the best. I'm at todaywasawesome. And if you go to my blog and make comments there, I don't read them. It's like archive only. I like keep it around because I worked on it 10 years ago. And every once in a while, I want to reference something that I wrote there. but no, follow me on Twitter and you can hit me up on LinkedIn if you like, or on GitHub, Slack, I'll usually get to. at some point, but I'm usually a little bit slower, but I think you had reached out over Slack and I actually went and checked Twitter three times to follow up on our conversation.

Bart: Thank you very much for your time today, Dan. It was great hearing all of your insights, uh, your comparisons, analogies, your fearlessness when it comes to putting out hot takes. or writing blog posts about fairy tales of Bitcoin, separate posts. But that being said, thank you very much for joining us today on KubeFM. I would like to ask one final question, is that if you could ask any question to the community, any challenge that you would like to see answered, what would it be?

Dan: Ooh, any challenge, any question answered. Hmm. Jeez, what a giant question. I don't know. I would like to see stateful Kubernetes reach its full potential. I would like to see that become easy to such an extent that people are like, stop saying that it's hard and they feel like, oh yeah, stateful stuff. I don't even worry about it. It just happens. It's great. That's what I'd like.

Bart: It's a challenge. It's been one that's been around for a while and has yet to be fully conquered. So I appreciate that. Having talked to a lot of people, having done a lot of live streams about that. So yeah, it's something that's still yet to be answered fully. And maybe 2024 could be the year. But in the meantime, we'll be in touch. Thanks a lot, Dan.

Dan: Thanks, Bart.

Bart: Cheers.