Hacking Alibaba Cloud's Kubernetes cluster

Bart: How can you hack Alibaba Cloud's Kubernetes cluster? We're going to show you how in this hacking episode with Ronen and Hillai, who are security researchers at Wiz. They have all kinds of skills when it comes to hacking vulnerabilities, which can happen due to lots of reasons. Among them, misconfiguration, which you'll hear about in this episode. They targeted a complex and feature-rich database, Postgres. From there, things got pretty interesting as they got access to a Kubernetes cluster. Let's check out the episode. What are three emerging Kubernetes or other tools that you're keeping an eye on?

Hillai: Ronen and I might know a lot about Kubernetes, but the funny thing is that we didn't really come from any Kubernetes backgrounds. We don't develop Kubernetes environments at all. We're regular hackers who went into Kubernetes hacking, not Kubernetes people who went into hacking. So we're not familiar with a lot of Kubernetes tools, to be honest. The Kubernetes tools we're most familiar with are those we found in our engagements, tools we were able to exploit to get ahead. So I think we're not the best people to ask about Kubernetes tools, but we're very excited about Kubernetes research, particularly. Very good.

Bart: Very good. What are three tools that you're a fan of?

Ronen: I wouldn't say tools, but maybe infrastructure, like service meshes. We are big fans of that. Also, from an attacker perspective, it's really nice to engage with now. And I don't know. We don't have any great tools in mind.

Bart: That's okay. Now, for people who don't know, can you tell me a little bit about what you do? I understand it's related to hacking and that sort of stuff. What do you do and where do you work?

Hillai: Ronen and I work for Wiz, a cloud security company. We are part of the vulnerability research team. In our day-to-day, we research major cloud services and providers like Azure, GCP, AWS, etc. We explore these services through their open bug bounty programs to find and report any flaws. Our goal is to make the cloud community safer, not only for our customers but for the community in general, by publishing our findings so everyone can protect themselves.

Bart: All right. Do you exclusively look at hacking cloud environments? Is this typical for security researchers, or are you in a niche?

Hillai: So I think it's not at all typical, actually. I think all of us, me, Ronen, and the rest of the team, come from backgrounds that are not cloud-related at all. We were familiar with hacking before we were familiar with cloud environments, but I think that much of the research we and others in the cloud security community do tries to evolve cloud security research forward. Finding new novel attack techniques and therefore making the cloud a little bit safer. That's what we try to do.

Bart: It may sound easy, but I don't imagine it is. In what way has your prior work focused on Kubernetes security? What have been some of the interesting things you discovered along the way, coming from an area that wasn't necessarily Kubernetes-focused but with a hacking background? What are the things you picked up throughout this journey?

Hillai: Many cloud providers use Kubernetes and container technology to manage their services. If I'm a cloud company with hundreds of thousands or millions of customers, it wouldn't be economical to set up a new virtual or physical machine for each one. Containers are an amazing improvement in how big companies can manage their huge infrastructure. Many cloud environments are built on Kubernetes. We focused on cloud and found Kubernetes in many of our engagements, not only with Alibaba Cloud but also with companies like IBM. We adopted Kubernetes because it's widely used. We didn't start with Kubernetes and then move to cloud; it was the other way around.

Ronen: I think the starting point was container security because we had many researchers focused on container security. We tried to perform container escapes and find new vulnerabilities that might affect containers, particularly container escapes. This led us to many other researchers who were using Kubernetes for their infrastructure. As a result, we had to learn Kubernetes and develop techniques to achieve our goals.

Bart: Reflecting on your experience as working professionals, if you could go back in time and share one career tip with your younger self, what would it be?

Hillai: Always follow your curiosity. When you're doing research, it's about following leads and hunches and doing things that make you curious. We were all a bit curious about cloud security because we were not from that field, and it started to emerge and become popular. We all got here because we were curious, wanted to find out more, and wanted to get into this new thing that nobody had done before. So, definitely follow your curiosity.

Ronen: I would.

Bart: And with that curiosity, following leads, and being researchers, what have been the best resources for you? The Kubernetes ecosystem moves very quickly. Have you mostly gotten informed through blogs, technical documentation, or watching videos? What has worked best?

Ronen: So for me, it was pretty much documentation. Of course, there are many blogs that I've read from cloud providers, like regular CNCF blogs, which contain really good information. The Twitter ecosystem, specifically for Kubernetes, is highly active, so I get regular updates from there about new technologies and new features. So for me, it was that.

Hillai: I agree with all that Ronen said. I would also add Reddit. I'm a big fan of Reddit. You have all these communities, security-focused communities, Kubernetes-focused communities, cloud-focused communities, and you get all this amazing content through there. I'm a big fan of Reddit for this.

Bart: Fantastic. Also in our case with the podcast, we've been seeing a lot of conversations on Reddit too, where people can really deep dive and take it as far as they want. But then as you mentioned, running about different resources as well, like Twitter, where a lot of conversations are happening, particularly in the Kubernetes ecosystem. It's definitely a good place to stay on top because, like I said, there's just a lot of information. So for today's conversation, in terms of our monthly content discovery, we came across an article about how you hacked Alibaba Cloud's Kubernetes cluster. And there was also a talk that you did at KubeCon. So, you're here today because of hacking this Kubernetes cluster. Might sound a little bit strange to say out loud. And here you are in a Kubernetes podcast, speaking about it. So before we dive into the details, tell us why. Why would you do that? And is this something that your employer was okay with?

Hillai: Our employer is absolutely okay with it. In fact, that's basically what we do at our employer. We always do cloud security-related research. Oftentimes, when you do this research, the best methodology is offensive security research. We try to act as attackers would when trying to hack these environments. Once we find something interesting, we report it to the vendor. By adopting this attacker mindset, we can prevent actual attackers with bad intentions from doing the same thing. This is what we do at Wiz. This is one of the engagements we did with one of many different cloud providers that service it.

Ronen: Of course, through those researchers, we always find some new techniques that we've never seen before. They haven't been published. So we always look at how to allow the community to see those techniques and understand how they can protect themselves from those things.

Bart: That's great. One of our previous guests talked about Kubernetes secrets management and threat modeling. How do you put yourself in the mind of the hacker that's going to try to exploit some kind of weakness? Is this something that you do on a regular basis? For example, when you're having lunch, do you think, "If I had to hack this restaurant, how would I do it?" How do you make that a regular practice? How do you go about understanding the thought processes of hackers to see how they view a system and how it can be exploited?

Ronen: I think most of our views on that come from the actual experience of those researchers. When we engage with some application, framework, or cloud provider infrastructure, we see those kinds of mistakes. We usually try to poke around and find those critical mistakes, the little things that people might overlook. From there, we try to understand the whole flow and how this could have been prevented.

Hillai: I would add an interesting point about that. When you're doing cloud security research, a lot of the times in traditional security research, when you're researching an application, what you want to achieve is RCE, remote code execution. You want to execute code on a machine, but in cloud security, because it's a wildly different world, remote code execution is often intuitive. You get a machine and you can execute your own code on it. So what's interesting to find? What we try to find a lot of the time when doing cloud security research is breaching barriers between different customers because this is the problem of the cloud. This is the unique problem of the cloud because you share your environment with hundreds of thousands of different customers. You want to try to prove, obviously without actually accessing any data, that you were able to access other customer data to breach that sort of trust that the cloud gives you. This is what we often do in cloud security, which is not traditional in security research.

Bart: With that in mind, when starting out this journey, how and why did you decide to go with Alibaba Cloud? What made that the fit for this particular case?

Ronen: We've actually started with the PostgreSQL research. We understood that many cloud providers allow you to use PostgreSQL on your managed offering, giving you an option to use it. We wanted to explore it further to see how top providers manage the infrastructure behind the scenes. We started to perform research across top providers. We found many vulnerabilities that allowed us to execute code on cloud provider-managed PostgreSQL instances. We also did a Black Hat talk about it. We tested big cloud providers to see if they were vulnerable to the actual exposed file, and through that, we also managed to test Alibaba and perform the research. Do you want to add something?

Hillai: As Ronen said, it started from Kubernetes and then expanded to Alibaba and various other cloud providers. If anyone is interested in checking out some of the other researchers, because everything is a little bit different, although they're all PostgreSQL, I highly recommend both the blog we did about PostgreSQL and the Black Hat Talk about Ronen.

Ronen: We also had a similar engagement.

Bart: The starting point was Postgres, not IBM. We have many links in our book, so you're more than welcome to check it out. In my experience, I've heard a lot about Postgres. It's a battle-tested, old-school database with a long history and a very strong community behind it. For the Postgres fans out there who might be worried that we're going to say Postgres is weak or easy to exploit, that wasn't really part of the criteria. It's not like we were evaluating this against MySQL or MongoDB.

Ronen: No, no, no.

Bart: What was the decision process like there?

Ronen: I think Postgres is a feature-rich database. You can do a lot of things with it, which is great overall. It also has many features that allow you to execute code on your database. With CloudRiders adapting this model to the cloud infrastructure, they try to prevent regular users from executing code on their managed instance and gaining system account permissions on their databases. We tried to bypass those restrictions and find a way to execute code on the managed database. It's not that the PostgreSQL project itself had those vulnerabilities. We actually exploited the modifications cloud providers made to their PostgreSQL engine to allow for the feat.

Bart: Now, to take this further, I spent a significant period of my cloud-native life creating content around running stateful workloads on Kubernetes. So, what is PostgreSQL's relationship to Kubernetes? I thought you had gained access to a Kubernetes cluster and not just PostgreSQL.

Hillai: Essentially, when you do these huge offerings for many customers, whether they're PostgreSQL or anything else that cloud providers offer, because it's managed infrastructure, the cloud is giving it to you and many other customers. Oftentimes, the most convenient way for the cloud provider to manage this is with containers, specifically with Kubernetes. That's the way for the cloud provider, in this case, Alibaba, to offer PostgreSQL in a very convenient, quick way to any customer that wants to respond. So that's the way they manage it internally. To you as a user, it's supposed to be completely transparent. You're not supposed to know that your PostgreSQL instance runs on this and that infrastructure. It's just a PostgreSQL instance, and you use it without knowing what's actually behind the scenes. But when we were able to execute code on that PostgreSQL instance using the flaws that Ronen just talked about, we were able to start to sniff around. We figured out that we were on a Kubernetes environment. But on the default installation that the cloud provider is giving you, you're not supposed to even know what sort of infrastructure it's running. It's just very convenient for the cloud provider to manage it like that.

Ronen: We've seen many cases where the actual infrastructure behind the scenes wasn't Kubernetes. Alibaba chose to use Kubernetes, IBM chose to use Kubernetes, but there aren't many cloud providers that had their own implementation behind the scenes.

Hillai: Got it.

Bart: One of the things we hear a lot in terms of security mindset is avoiding vulnerabilities related to misconfigurations, which often stem from human error. What were the main misconfigurations you saw that were leading to vulnerabilities?

Hillai: In this case, I think the main misconfiguration we found is that containers shouldn't be considered a security barrier. And definitely not the sole security barrier. It can be a security barrier as part of your entire installation environment. But containers by themselves are not strong enough to separate me, the Eli customer, from the Ronen customer or the Bart customer. These are supposedly three different companies that aren't supposed to know anything about each other. We think containers are not strong enough to do this because every time there's a kernel vulnerability in Linux, which is not rare, it immediately bypasses every protection because the kernel is supposed to govern this. Misconfigurations are also very rampant, like the misconfiguration we were able to exploit in this particular research. We would not recommend containers as security barriers. Other flaws we found include very strong secrets within the Kubernetes environment. These secrets not only allowed overarching read access but also write access, meaning we could overwrite packages distributed to many different cloud services and customers in Alibaba. When creating these strong secrets, you can allow a person to travel between different environments, services, and customers all through one secret. That's definitely something we would not recommend people to do.

Ronen: To add to Hillai's point, I think the secret he's talking about is the image pull secret, which is a really... core concept in Kubernetes. Basically, when you want to pull images from a private content registry, you have to use this image pull secret to configure your registry for the network. If you don't configure it correctly, you might put a secret key that has push permissions. This key should only have pull permissions. If an attacker gains access, like we managed to get in Alibaba... This can be devastating for your environment.

Bart: I think for many people without a strong security background, they might assume that security experts like the two of you come in, click a button, it scans, and finds these vulnerabilities. I think in the security world, like many other fields, it's a combination of art and science. You talk about the creativity necessary to find vulnerabilities or misconfigurations, whether related to permissions, namespaces, or things of that nature.

Hillai: I definitely agree. There's a lot of creativity to security research. Oftentimes, when you read about a novel new attack vector that somebody finds, it's just creativity. Just think of something that nobody else has thought of before. In this particular research, when we started to look for patterns of things that we already knew were dangerous, like hypermissions and shared volumes, we didn't find them. That was a moment of, okay, now we should stop and think because we hit this wall. We couldn't find anything interesting within the container. So we had to think outside the box. What else can we do to make this container do something interesting? At this point, we went out of the container, out of the code execution that we were able to find. We went back to the control panel that Alibaba Cloud gave us and started clicking on all the buttons we could find to try to make something interesting happen. This was one of the big breakthroughs in this research. We found this button that enabled SSL encryption for the Postgres SQL instance. Once we clicked that button, lots of interesting stuff started happening within the container. We were able to follow the new activity. There was new stuff that we hadn't seen before. Let's follow that lead. That was the lead that allowed us to escape the container. Like you said, a lot of times it's just creativity. We tried all these well-known patterns, and none of them worked. None of them were misconfigured. So we had to find new features, new buttons, something new that we could do. This is what led this research to success, to container escape.

Bart: With that in mind, moving forward, can you inform our audience about what SCP is, what role the SCP command played in the attack, and how it was exploited?

Hillai: SCP is a file transfer utility that's in every Linux box, basically. It uses SSH connections to transfer files between machines. In this particular case, the SSL encryption feature we clicked on worked by using a new container, an Alibaba management container, which ran the SCP command on our container to move the SSL certificate. from here to there. We exploited the fact that the SCP command read its configuration by default from a directory we controlled. We added our own malicious SSH configuration that, when loaded, would run a command. This allowed us to move from our weak container to the Alibaba Management Container because it took the command from us and ran it there. We were able to skip from here to there, which was nice. It's a nice little trick.

Ronen: One of the most critical points we had was that the shared volume was essentially a shared home directory. We had the same user in our container and the management container. So we shared the home directory. SCP, by default, reads your configuration from your home directory. By changing this file to a malicious one with our command, and because it was shared between the management container and our container, we could exploit it.

Bart: What does the success of creating a privileged container via the Docker API signify about cloud security more broadly?

Ronen: Many Kubernetes environments use Docker as their container management component. When you manage to get access to the Docker API socket, for example, through an HTTP request, you can spawn a new container, possibly a privileged container, which will share many namespaces and possibly also volumes with your host. The host is the actual Kubernetes node. If you spawn a privileged container with all those components, you basically achieve Kubernetes node access, allowing you to access everything that the node has access to.

Hillai: From the guest to the host. From the pod to the host.

Bart: And what was the next step when you gained access to the underlying Kubernetes node? Because it shouldn't give you access to the entire cluster necessarily. But only a limited view. Yes.

Ronen: Now that you have code execution on the node, you can use the Kubelet credentials to start poking around, maybe to get commands, codes, secrets, and other information. But on a default Kubernetes setup, you shouldn't have access to all those resources. Alibaba configured their Kubelet credentials to have those permissions, and using the Kubelet service account, we managed to get broad access to the whole cluster. We could list all codes in the cluster, which contained customer information. We could also use the get secret command to retrieve all secrets from the whole cluster. This was a significant misconfiguration that allowed us to see all those resources.

Bart: And this is all still on a single node, correct?

Ronen: We were running on a single node and used our kubelet credentials to see all the other nodes in the cluster and the resources as well.

Hillai: Although that node was separated from other customers and only belonged to us without containing any other customer data, its service account, the Kubelet account, was too powerful. Although that specific node was isolated, its service account was able to access all this sensitive information, including pods, nodes, and secrets belonging to other customers.

Bart: All right, so you took over Managed PostgresQL, offering it to Alibaba. What happens next? You call up Alibaba and say, hey, we have a story to tell you. If you start using free Postgres for your personal projects, what was the next step after that?

Hillai: So definitely not using the free Postgres. Every time that we, at this point in any research, can prove that we're able to access other customers' data, then it's definitely game over. We don't want to get into a situation where we accidentally view other customers' data. At this point, we gather everything that we found. We send reports to Alibaba Cloud, and they responded very quickly and professionally. They updated us on all the fixes they deployed to address the problems along the research pipeline. Everything that we find and deem a critical issue, we report immediately so that others can't exploit it for other purposes.

Bart: Did they send you any swag to say thanks? Or did they, maybe not so much that, but can you tell me about any of the fixes they may have implemented to address the issues that you brought to their attention? We found actually a couple of big mistakes.

Ronen: The first issue was the false growth issues that we found and used to exploit code execution on the OS. We worked closely with them to help fix these issues. Secondly, they fixed the SCP problem, which allowed us to move from our container to their management container. They fixed that. They also restricted the Kubelet permissions to only scoped ones, not broad ones like they had before.

Hillai: They limited the image pull secret permissions to have only read access and not push access. Every flaw that we were able to find along the way was fixed separately. In addition to that, they also implemented some sort of safe container technology developed internally at Alibaba. Similar to Google's GVisor project, which is an open-source container hardening technology that makes the container more secure and harder to escape. They implemented that as an additional safeguard in addition to addressing all the issues we found.

Bart: Throughout this process, what would you say were the key lessons learned?

Hillai: First of all, the containers are not a strong enough security barrier. They can be bypassed in a myriad of different ways. Obviously, every security barrier can theoretically be breached. But we feel like containers are not a high enough bar for breaches and for isolation between customers. Containers should never be the sole security barrier. They can be a security barrier, but there have to be additional safeguards in place. We would like to build a defense in depth to ensure that a singular vulnerability won't allow my company to access the data of a competitor company. That's one thing. The second thing is the management of strong credentials. As Ronen said, Alibaba originally had a very strong secret that could read and write data across the environment to an image pool secret that could push new Docker images to the central registry and spread to other cloud services in Alibaba. Following a report, they scoped those credentials so they wouldn't be that strong. We advise being very careful with strong secrets like this. Ideally, not having them at all and having each strong secret scoped to a specific action. When you have a strong secret like that, it can help people hop between environments, production environments, development environments, testing environments, and development workstations. Regarding the container itself, the first vulnerability, the SCP vulnerability, another lesson is the namespace sharing between containers. In this case, Alibaba spawned a new container that interfered with our container, a customer container, so it shouldn't be trusted. It had a shared namespace and a shared home directory, which you were able to exploit using the SCP trick. We advise being very careful when sharing namespaces between your own trusted container and the customer's non-trusted container. Be extra safe about everything you share and never give any extra permissions unless absolutely necessary. Every lenient misconfiguration can potentially be exploited by attackers.

Bart: For people in our audience who may want to have a further conversation with their boss about which tooling to keep in mind... We talked about this in the beginning regarding which tools are catching your attention. To implement some of these tactics, are there any tools that you would recommend that people may not be aware of?

Hillai: There are lots of different tools. I'm sure there are even tools that we're not aware of that are doing a great job because there is a very active Kubernetes community. What I highly recommend is a framework called Peach. It's an open-source project that we developed with a research team. It was developed, it's completely open source, and it's made possible because of contributions from lots of wonderful people from many great companies. Basically, it's a framework for how to build an isolated environment in the cloud or not in the cloud. To truly make the fence in depth when you're serving things to different tenants, different customers, and you want to isolate them properly, it's basically like a white paper resource on how to do this correctly. What sort of mistakes should you be aware of? What sort of things should you be on the lookout for and how to correctly implement those safeguards. We highly recommend checking it out. I think it's peach.wiz.io and it's also on GitHub. I highly recommend checking it out if you have your own multi-tenant environment or your own environment where you want to isolate things from each other. It's a great resource, and if anyone listening to this knows any tricks or tips that we don't, contributions are also greatly appreciated.

Ronen: I would also recommend some secret scanning tools. In all our research, we usually perform a secret scan.

Bart: On our machine on the cloud infrastructure, we always find some secrets through that. Environment hygiene is really important, especially in managed environments. There are many good tools for secret scanning, so we highly recommend anyone to use them to make their environments safe. Since the subject of multi-tenancy came up, the attack that you're describing today for our audience originated from a common practice of sharing a Kubernetes cluster with multiple tenants. Given the gravity of the flaw and its potential consequences, is there any recommendation for securing multi-tenant clusters in Kubernetes that you'd like to share?

Ronen: I think the most important thing when implementing multi-tenancy in Kubernetes can be separated into a couple of things. First is network security. By default, Kubernetes doesn't prevent you from accessing different nodes through a network. So implementing good network isolation is a must. Secondly, if you are already talking about multi-tenancy, separating namespaces between customers can be a good approach. Implementing good container security technologies, like GVisor or Kata containers, and not relying solely on the security of Docker to prevent container escapes is important. I think it's a good approach. Those are the key points I would focus on.

Bart: Particularly with the part about containers, in terms of hardening containers so they're more difficult to escape. Any recommendations or advice you'd give there?

Ronen: I can give you an example from the Alibaba research. They had Linux namespaces shared between containers. For example, from our management container and our container. When you design a system that shares namespaces or resources between management containers and regular user containers, you always have to assess those risks and be aware of them. Sharing Linux namespaces can be dangerous. Using container technologies like GVisor and Kata Containers can potentially prevent you from Linux kernel vulnerabilities that attackers might exploit in your environment to achieve kernel-level code execution, allowing them to hop to the Kubernetes node.

Bart: A question that I want to ask here, for a lot of our audience that might not have such a strong security background, is: Is everyone a security stakeholder? Are there certain things that just can't be overlooked by any Kubernetes engineer because of the environments in which they're going to be working, the risk, the danger? Is it just hyper paranoia? What do you think about that? Is security something that should be siloed, or should it be shared across an organization? Is this something that people really need to make sure is available in their tool belt? What sort of advice would you give for regular Kubernetes engineers that are out there who may not have much contact with this area?

Hillai: So I think that security is super important. You only have to open the news these days to find out the sort of laws that everything from small startups to huge corporations can fall victim to, you know, the actual bad guys, not just white hat security researchers, almost every day or every week. What people have to understand is that when you're creating a service that is on the internet, it immediately becomes a target to all sorts of actors in the cybersecurity space, whether they are white hat hackers looking to make a little bit of money or people that do ransomware, data stealing, and other malicious activities. Even the smallest project doesn't have the... luxury of not caring about security. But that's on the bad side. On the good side, there are a lot of tools today that help people achieve these goals without doing a lot of manual work. Tools like GVisor that Ronen mentioned, which is a project by Google to harden containers, are not extremely hard to implement because you don't have to write them yourself. By implementing these sorts of things, you get added protection. So I think people should be interested in that. Every project, community, and company should be interested in securing their systems from external attackers. The good thing is that there are a lot of tools to help you be secure.

Ronen: I think it's not only tools but also a lot of resources online that you can read about.

Hillai: Documentation, blogs, which could actually help you.

Ronen: Actually help you understand those risks and help you protect them. Kubernetes by itself has many good features, such as security, default security policy, and many other things. So we encourage people to be aware of security in general and make their environment secure.

Bart: Fantastic. In this case, you discovered this vulnerability problem and notified Alibaba. But what's stopping you from not joining the dark side and thinking, I'm aware of this vulnerability and I'm not going to let them know about it? Do you think that Alibaba would eventually catch on?

Ronen: I'm pretty sure because right in the middle of our research, they already started taking steps, which they told us. I think every cloud provider has their security teams on it. They are always monitoring their environments; they know you're there.

Hillai: I think cloud providers are doing a great job. From our side, we are white-hat attackers, so our goal is to always make the community more secure and make the cloud more secure. Offensive research is just a tool to do that. We want those vulnerabilities to be fixed. We love hearing that when we report something, a new product update was launched to many customers that helped them be secure. This is the reason we're doing this: to help these products be more secure and to help customers learn how to secure their offerings. That's why we... Publish these blogs and talks, so that practitioners from the defender and developer side can see these things and realize, "That's actually a mistake I could also be making in my own environment." This is basically the reason we're doing this.

Bart: Very good. That being said, what's next for the two of you? Who are you going to attack next?

Hillai: We constantly work on new stuff. We recently published a blog about Hugging Face, the AI provider, by Sagi from the team who is not here at the moment. We are constantly working on new things. There are a lot of researchers currently under disclosure that we can't disclose until everything is fixed. Definitely keep an eye on our blog. That's the first place we post when something interesting drops. We definitely have good stuff in the oven. Stay tuned.

Ronen: Of course, from those resources, many Kubernetes researchers will also come.

Hillai: Hugging Face, by the way, is definitely a Kubernetes researcher. If anyone wants to read something that is actually news and very new, I highly recommend the Hugging Face blog.

Bart: Good. If people want to get in touch with you, what's the best way to do so?

Hillai: So we're all on Twitter. My handle is @T-L-I-H-I-L-L-A-I. And mine is @Ronen.

Ronen: SHH.

Hillai: We're very active on Twitter. You can DM us anytime. You can also email the team at [email protected]. But I think Twitter is where we are most active. Fantastic.

Bart: I'm sure people will be getting in touch with you about this. It's one thing to do it, but how you share your experience so eloquently and get into all the details that you said for a lot of Kubernetes practitioners. Simply aren't aware, they might come from one background or another, but not assuming that this is going to become part of their daily lives. Well, guess what? It's 2024 and it very much is. Like you said, all we need to do is read the news to know about that. Thank you for the work that you're doing. I think it is a very valuable service to the community, even though some of it may be behind the scenes. It's your job so that it doesn't become a problem that's exploding right in front of somebody, but rather something that can be maintained in conversations directly with cloud providers as you've done with Alibaba. Thank you very much.

Hillai: Thank you.

Bart: Wonderful having you and look forward to seeing you whether it's in KubeCon or online. Take care, you guys. Cheers.

Hillai: You too.

Bart: Thank you.