The ticking supply chain attack bomb of exposed Kubernetes secrets

Bart: Today on KubeFM, we're diving into the critical roles and responsibilities of cloud-native engineers in threat intelligence and offensive research. To do so, I spoke to Yakir and Assaf from Aqua Security, and we explored the significance of staying current with industry changes and the vital importance of Kubernetes secrets in preventing supply chain attack threats. Our discussion highlights the severe impact of leaked credentials, emphasizing the need for security awareness and the use of scanning tools to prevent data leaks and those pesky misconfigurations that can cause a lot of problems. We'll also touch on the importance of encrypting secrets, developing a mature threat model, and the impact that crypto mining attacks are still having on Kubernetes clusters. Check out this episode as we share insights on making security more accessible to developers and all stakeholders across cloud-native organizations to enhance cybersecurity for all. Now, supply chain security requires visibility into the core of the system. Naturally, eBPF comes to mind. And eBPF-based runtime security equals Tetragon. Tetragon can look deep into the system, connecting network data to the actual binary that caused it for deep workload visibility. It also analyzes real-time TCP/UDP metrics, understands enforcement mechanisms inside the kernel, and overall ensures compliance across the clusters when needed. NIST, anyone? Follow the link in the description to discover how Tetragon by Isovalent elevates Kubernetes security. Now, let's take a look at the episode. To get started, we want to know which three emerging Kubernetes tools you are keeping an eye on.

Yakir: Yakir, we'll start out with OPA Gatekeeper, Kyverno, and Kubewarden. I am a huge fan of defining your own policy and securing your Kubernetes cluster by setting different rules based on your specific needs. There are great rules that already exist in these projects. We are currently researching different rules in this area and trying to improve them because we found some misalignment or problems with some of them. We would like to suggest in the future some ways to secure them and secure your Kubernetes cluster with better rules.

Bart: Do you agree? Any other tools that you'd like to look at?

Assaf: I completely agree with Yakir. I would like to add one more tool, which is Tracee. Tracee is developed by Aqua, so I have my own interests here. But to be honest, when we work with Kubernetes, we really have a mindset of defending the Kubernetes cluster. Tracee really helps us by monitoring the eBPF level of the Linux servers, all the syscalls, and all the network activities. It's a really good tool from a security perspective.

Bart: Very good. Since you mentioned Aqua Security, it ties well into our next question. We know that you work at Aqua Security. What do you do?

Assaf: I lead the threat intelligence operation, which is basically, we are a security company. We need to be aware of all the attacks, everything that is out there, what the attackers are doing, and how they are circumventing Kubernetes clusters and Linux servers. This is what we do. We look at the software development lifecycle and understand it. My role is to invite everyone to attack our honeypots and to learn from that. Very good.

Bart: And Yakir, what about you?

Yakir: I am more on the red side, and by that, I mean I'm researching more offensive techniques on cloud-native technology, the cloud, CI/CD, supply chain, and different areas. I try to think about new attack vectors and what attackers might do, and report our findings to some vendors before attackers can exploit them. Sometimes, we introduce or present our findings at security conferences or think about solutions, sometimes in our product, and sometimes provide solutions for the open-source community. We also aim to increase awareness among developers and DevOps around different areas.

Bart: Speaking of the open source community, how did you get into Cloud Native? What did you do before that, and what started the change?

Yakir: I was a penetration tester and wrote emails that mostly worked on on-prem networks and different technologies. I have experience in mobile, VoIP, and various technologies. After some time, I felt that I wanted to move on to something more popular and increasing, which is actually cloud-native technology. I decided that I wanted to understand this technology more, and in my opinion, there aren't many researchers in this area. I want to be one of those who try to find new risks before others do.

Bart: Good. What about your case, Assaf?

Assaf: What about you?

Bart: How did you get started in cloud native?

Assaf: So it's very similar to Yakir. I started in security, e-fraud, commercial fraud, and then I went to threat intelligence. I led a group of threat intelligence analysts, and then I thought it would be very interesting to go deeper into technology, and cloud native is the best option. So I joined Aqua Security and since then I've been working with the cloud-native environments, understanding threat intelligence landscape in these areas.

Bart: It's no secret that this is an industry that moves very quickly. What have been the best ways for you to stay up to date with all the different changes that are happening? What are your favorite resources, whether they're blogs, videos, or tutorials? What do you follow?

Yakir: I follow some subreddits about security. There is a great subreddit called Netsec and a Kubernetes subreddit. Sometimes, after conferences like KubeCon, I like to read briefings from other reviewers and watch some of the recommended sessions.

Bart: Anything to add, Assaf?

Assaf: Yes, to be honest, a lot of documentation is available on kubernetes.io. There are also online courses with excellent labs, and I guess that's the best way to get updates.

Yakir: I think that from the honeypot we get many insights about defensive strategies.

Assaf: That's the offensive side.

Bart: Regarding the honeypot, for people who might not be aware, how do the incentives work for people to try it out? Let's say someone's never done this before. What are common questions you get from people who are curious but aren't sure where to start?

Assaf: First of all, you need to read a lot because it's very easy to make mistakes. If you don't know what you're doing, you might create software development lifecycle environments, like CI/CD, applications, or a Docker API, and then expose them to the internet with vulnerabilities or misconfigurations. If you don't know how to segregate and prevent lateral movement to the cloud, you can end up with hundreds of thousands of dollars missing from your bank account because attackers created a crypto mining operation. This is something that you really need to know what you're doing. There are some good Docker in Docker options and environments to start with, but... Don't get into it unless you really understand what you're doing. Once you understand how to build strong environments to prevent lateral movement, then it's really nice to see. Some specific applications or honeypots. You can look for VulnHub; there are some good misconfigured and vulnerable applications there. Once you upload them, you need a monitoring tool. That's why I mentioned Tracee, because you can monitor with eBPF-based technology, the syscalls, and the network, and then you can understand what the attackers are doing. You can monitor the network. To understand what areas they are targeting at the Kubernetes level. But I wouldn't recommend starting with Kubernetes. Start small with Docker in Docker, which is a very good start.

Bart: Regarding your journey as cloud-native engineers, particularly in the security space, if you could go back in time and give yourself one piece of career advice, what would it be?

Yakir: Start reading and watching materials as soon as possible. Even if you don't understand all the details right now, each piece of information could serve you in the future, and this way you build knowledge. You do not need to be afraid of technology like Kubernetes. Just jump into the water, read, and watch things. Eventually, things will make sense, and you will understand them.

Assaf: I totally agree with Yakir. There are some good tutorials, especially on YouTube, that teach you how to build things from scratch. For example, you can use EKS to create a Kubernetes cluster, but you can also build it component by component and connect them together, which is a good way to learn and understand by working with your hands.

Bart: Now, as part of our monthly content discovery series, we found this article about the ticking supply chain attack bomb of exposed Kubernetes secrets. We've talked about Kubernetes secrets before on other episodes in the podcast, particularly with Mac Chaffee. We wanted to explore this topic further. Today, we're going to be focusing on the critical issue of exposed Kubernetes secrets and the potential supply chain attack threats they pose. Before diving into the findings, could you explain what Kubernetes secrets are and what they're usually used for?

Assaf: Kubernetes secrets are one of the basic components of Kubernetes clusters. Kubernetes is not a standalone system. You need to pull container images from somewhere. You need a secret or a token to access your registry. You might want to connect it with your organization. You may need your SSO, SAML token, or a network TLS certificate to add to Kubernetes. This is how you can connect it to the software development lifecycle and your organization. Kubernetes needs to store it somewhere and know when and how to use it. This is how Kubernetes secrets come to life. It's a structured way to allow communication with a secured environment.

Bart: How many different kinds of Kubernetes secrets are there?

Assaf: Okay, so officially there are eight based on the kubernetes.io. It's kind of an arbitrary and logical decision, these eight. You can have five or you can have ten, but it's basically a way that Kubernetes validates these secrets and sees that they're not broken and they're complete. So there is a reason for this division. But if you want to go a little bit deeper... Let's review them. Let's say you have a node and you want to connect it to your Kubernetes clusters. You need a secret for the kubeadm to connect it because otherwise, you would be able to connect any node in the world. So you need a secret for that. You have your service account, so you want them to do something. You want them to connect with the API server, so you need another secret for that. If you want to do SSH or have TLS, you have secrets for those as well. And I mentioned the registries. You have two secrets for that. You have docker config and docker config JSON. And last but not least, which is the mother of all, you have the opaque, which is basically any secret you want. If you want to connect Postgres or Grafana or whatever you want to connect, it's basically like anything that you want to add. So again, it's a little bit arbitrary because with the opaque, it's like endless possibilities.

Bart: If there's one piece of advice when I think about secrets, it's from the cloud native wizard Gandalf in the Lord of the Rings, who says, "Keep it secret, keep it safe." We're thinking about safety. If secrets are meant to be secrets, how do they get leaked out into the wild? How does that happen?

Yakir: I think it's a classical human error, and the problem with Kubernetes is not the secrets. We need secrets; they have their purpose in the server and the program. The problem is that if we look at the development lifecycle, when someone uploads it to public GitHub or GitLab, it's a problem because attackers could hunt these specific secrets of Kubernetes or other secrets. They can actually write specific regex to find any YAML file that contains a Docker configuration and then use regex to hunt for base64-encoded secrets. In this case, attackers could find them, but we need them. Later, we will suggest some mitigation strategies. If you want to upload your secrets to a platform like GitHub, you can do it, but you need to do it in a more secure way. One of the problems we have seen in this research is this confusion between encryption and encoding. People look at these strings and think that it's okay because it's encryption. Even if attackers find this, they will not be able to decrypt it. But as I said before, it's encoding. So I can always decode this to the original plain text.

Bart: Are there any insights regarding what kind of projects, the kind of companies that might be attacked, their size, their maturity level, what kind of projects are leaking those credentials, and where those registries are located?

Assaf: So basically everyone, because as Yakir said, and I completely agree, it's a human-computer interaction problem. In all companies, there are humans, and in all companies, there are computers, and they interact. So, it's a global problem for all companies. We've seen companies like Microsoft, companies from all sectors, from the financial sector, which is often more secure than others, from insurance, from automotive, even the security industry. These sectors, which are supposed to be more secure, also have these problems. Just to emphasize the problem, think about when you have contractors and you want to share registry secrets with them. Sometimes they use their own GitHub account and, by mistake, upload the secrets. Think about open-source contributors or open-source engineers. By mistake, they expose their entire Docker Hub or the entire registry. When they do that and the token is privileged, anyone can get into the artifactory or the registry and control it. When they control it, it opens a supply chain to everyone. That's what we mean by human error. There was a highly respected open-source engineer with many stars, and we found their Docker Hub registry credentials exactly in this way.

Yakir: I think he was a security advocate too.

Assaf: And we showed it to him. He asked to have a video call. And he told us, "You know what? I'm ashamed, but it shouldn't have happened. It's a mistake. And obviously, it could happen to anyone in any rank, in any kind of industry, in any company. That's human nature, unfortunately. But that's why we need good controls and good security to prevent it."

Yakir: We actually published a follow-up blog for the supply chain blog on Kubernetes. We disclosed companies like Azure and Reddit where we found their secrets that could lead to major supply chain attacks. For example, in the case of Azure, we found a token for their Azure internal registry that they were using for some of their internal projects. We reported this finding to them, and they quickly revoked this token. If an attacker had accessed or used this token before us, it could have led to catastrophic results for Microsoft, Azure, and their users.

Bart: And the thing is, I think a great takeaway with this is that, like you said, not just startups say, "Oh, security, we'll deal with that later." And then they're completely exposed. We're talking about large-scale enterprises that have significant security teams. But as you mentioned, this human error, and some projects that are being actively maintained and others in terms of how often they're updated, there are a lot of things to keep in mind here. So when you find these credentials, what do you do? Do you pull enough images to trigger the Docker Hub limits? Or what's your approach?

Assaf: So obviously we run somewhere the organizations and try to bring them to the news. No, what we do is we have a very strict protocol called the Good Samaritan Program. We really try to locate the organizations. I just came from a conference in Amsterdam and learned that the Dutch government imposes on all Dutch organizations a strict policy to add security.txt on their websites. So that you will be able to get all the information and contact details of organizations. It's really important because many organizations don't have this address, and we should adopt this Netherlands policy. That's how you can contact organizations. There are many organizations, especially smaller businesses, which you just can't contact.

Yakir: We gave up because we tried to find employees of these companies, left a message for the CEO, and didn't get any response.

Bart: So it's important for companies to make themselves available and accessible so that when these things come up, they can be alerted. Someone in the security industry told me a long time ago, there are two types of companies: the ones that know they've been hacked and the ones that don't. As much as that might sound like marketing for security companies, I think there's a significant element of truth there. Companies need to make it easy for them to be informed that these things have happened. I think that's a good point. In terms of how these leaks—it's not just Azure, it's all other users, whether it's Red Hat or whoever it happens to be—how these leaks could affect various organizations from small businesses to large enterprises. Are there any specific cases that you found to be surprising and that you might want to share?

Yakir: I think there are some scenarios. For example, sometimes we found a private token for Docker app, and when we logged in, we found Docker images that we can search for secrets and other sensitive information. In this way, we expand our attack surface and get even more IPs, domains, secrets, and source code. It would serve us if we were attackers to expand the attack on the organization. Sometimes, if we have the right access to a Docker registry, we could replace or backdoor some of the existing images. In this way, when someone pulls this image again, there will be a backdoor, and we will get initial access. We could then start something called lateral movement to see how we can expand in the network and find other valuable assets.

Assaf: I saw a very interesting tweet a few days ago. It says, hackers don't hack anymore; they just use the keys. I think it's a very good way to emphasize and support what Yakir is saying. These keys are actually the keys to the kingdom in some ways.

Yakir: Why bother trying to unfold vulnerability or complicated payload if you could use a key that actually... I will connect during regular working hours and use the regular API call to pull another resource cell. I hope to remain undetectable in my campaign as a package.

Bart: One of the things to keep in mind is the mentality that organizations need to have. It's not just about constantly looking for weaknesses to exploit. It's about doing things as if you were a regular employee, like connecting during working hours and using the keys, et cetera. You've notified organizations about the things they need to be aware of, but others who get this information might not be so altruistic or friendly. What's the basic advice you give organizations about how they can prevent scenarios where keys are available? What can organizations do to be safe?

Yakir: First of all, I think some of them already do this, but run a secret scanning tool, and not just one, run several of them, because each one is different. Some of them are based on Regex and some are entropy-based, so you will get totally different results from different tools. As we suggested in our last blog, we would like organizations and employees to scan their own private repos because mistakes happen and people sometimes upload secrets to their own private or public repository on GitHub. We need to check those repositories too, because in this research, most of the secrets we found were in personal employee repos and not in the company's official repo.

Bart: We can say that in many of these cases where secrets were leaked, it was because they weren't using scanners.

Assaf: Well, no, not necessarily. The direct cause of most of the secrets that we saw was shadow IT in the cloud-native environment, such as personal GitHub and personal registries. These weren't monitored by the company's tools. Most companies that we work with are using a threat model. In the threat model, you need to raise awareness, add that to the policy, and understand that shadow IT is an issue. In cloud-native environments, add these environments, such as GitHub, GitLab, Bitbucket, and so on. If they understand that and add it to the threat modeling, and use the proper tools like Yakir said, that will cover these risks. Shadow IT is a huge problem. It's not a new problem; it's a huge problem. It just came to the cloud as well, and they need to pay attention.

Yakir: I would like to add something. If you still need to upload secrets to your public GitHub repository, there are great tools that can help you encrypt these secrets. For example, a sealed secret or a Mozilla SOP that can encrypt your secrets, allowing you to upload them to your GitHub account. When you want to apply them with kubectl, the Secret manifest in sealed secret has a controller with the key that will encrypt these secrets. This could help you in the case of Kubernetes, allowing you to upload only encrypted secrets to GitHub if you want to share a manifest for some reason.

Bart: Just to go back to the specific cases that you may have seen, because I know in the beginning you mentioned the risk of crypto mining happening. This is something we've been hearing about for, I would say, two or three years. Is there a particular set of risks that blockchain companies need to be aware of, or is it an industry that might be a little bit more on the wild side? Any comments regarding that?

Assaf: The crypto mining aspect is, these are two separate things. This was very peculiar because we did see a lot of blockchain companies that had this problem more than other industries. We are a little bit puzzled about that. But this is one threat vector or attack vector. What you are saying is the impact. We do see a lot of crypto mining attacks because it's a low-hanging fruit. You get access or remote code execution on a server and you want to exploit it. The immediate gain for financial attackers is to crypto mine because you get money from that. The next gain is to sell the secrets, to go laterally and attack the cloud account, maybe the S3 bucket, and to gain more secrets, to gain more access to other environments. But this is a more direct way to gain money.

Yakir: And you can automate this. The second way will require time. In the first approach, you can actually automate this and find an open instance. Assaf is the expert in this area for sure.

Assaf: We see a lot of attacks. There are many vectors. There are many ways to do that. This is one thing that you mentioned. The second thing, and I started with it, is basically the blockchain companies. You want to target these because then you get access to ledgers and wallets. Instead of doing all the hard work, you just steal the wallet, which is like a credit card or another thing. Blockchain companies should be aware of that. It's a warning sign for them, or a red flag, to be aware that it shouldn't happen to them. They shouldn't leave their secrets exposed in the open.

Bart: With things that we're thinking about, ways that companies can be equipped to deal with these problems. In terms of preventing leaks, scanning and mitigating misconfigurations in Kubernetes secrets. What are some pieces of advice that we could think about here when it comes to data encryption or regarding privileges, expiration dates? What are things that you recommend regarding those?

Assaf: In our research, we saw a lot of things. Yakir also mentioned that. In the Kubernetes threat model, you have encryption of data in transit and data at rest, which is basically on the Kubernetes cluster itself. Yakir also mentioned encrypting the secrets when they are outside of the Kubernetes cluster. We've seen a lot of cases like this, where you just can't do anything. When it comes to GCP, for instance, they have expiration dates, also AWS, but GCP is more limited or shorter. When you try to get access to the GCP registry, for instance, it is outdated and you can't do anything about it. Maybe having shorter expiration dates like the GCP model using two-factor authentication. For instance, some of the Docker Hub secrets we found require two-factor authentication. There are ways to circumvent two-factor authentication and exploit that. So it's not completely bulletproof. But for attackers... script kiddies or those just trying to get the low-hanging fruits, they will give up, and it's another way to protect your accounts. Yakir, do you have any further tips?

Yakir: I think maybe using vaults, HashiCorp Vault or KMS vaults of different cloud providers, because they provide better observation on your secrets. This way, you can verify and rotate them. I think it is a better approach than using plain text or encoding secrets in your manifest. As we said before, always scan your code, not just once. Set up actions in your CI/CD for checking secrets. Even if you do not find secrets, rotate them after some months. Assume that someone already has them in some way. This way, you can prevent or minimize the risk. If we talk about defense in depth, you need to secure your cluster itself. Even if an attacker gains access to a specific pod or node in your network, you need to minimize the risks and attack surface by isolating them as much as possible. For example, separate Kubernetes cluster environments: the dev environment, the production environment, and the staging environment. If one of them falls, it will not affect the others.

Bart: One of the things that has come up over the years regarding misconfigurations is the idea. I'm curious about your thoughts on this: everyone is a security stakeholder in the organization to some degree or another. When we're talking about development teams, what should they keep in mind during the development process? What are security recommendations that you might have for teams in that case?

Yakir: I think that developers need better interaction with researchers. I read that there is one researcher for around 100 developers. Many developers are not aware of different risks, do not read articles, and do not know threat modeling. We need to increase awareness. It really depends on the company, in my opinion. Some companies have programs or try to encourage employees to think in a secure way.

Assaf: I completely agree with you. It's all about the human factor, as we said, and the weakest link. All you need is one engineer, one developer who makes a mistake. I was on a call a few months ago with 20 security engineers because we disclosed the problem to them. We told them, your Kubernetes API server is exposed to the internet with anonymous user access and with privileges of admin on it. It was a huge company that can impact the lives of millions of people. The senior VP of security told us after the aftermath, the problem was that one of the security engineers or one of the DevOps didn't thoroughly understand the AWS model. He exposed the API server to the internet and added admin privileges. He didn't really understand the consequences or the impact. And that's it. Just a simple engineer doing that. So coming back to Yakir, you need education and training. You need to take baby steps if you're new to the organization, just to understand what we're doing. You need some controls to ensure that your Kubernetes cluster is not exposed, that you did not make this or that misconfiguration, because there are millions of misconfigurations and they are not controlled by NIST or by a governmental organization. They are not documented as well.

Bart: In the article, you mentioned some specific points regarding GitHub's secrets management tool. I know you've referenced those before. Is there anything else that you would recommend?

Assaf: No, I think Yakir spoke about it, like HashiCorp and using AWS KMS to encrypt your secrets. I think we mentioned all that throughout the conversation. I don't have anything new to say. But it's something that you really need to be aware of, I guess awareness and...

Yakir: Maybe prevent secrets from uploading to GitHub from the beginning. There is a feature called Reux that could alert you about secrets that have been committed to GitHub. Some companies actively scan for leaks, and GitHub has added a feature with specific partners to inform you if they detect any leaked secrets in your repository. However, the user needs to actively enable this feature. We definitely recommend activating any security feature available. And I think that's it.

Bart: That's good. Since you mentioned HashiCorp, one controversial question. If Kubernetes secrets are not so secret, should they be abandoned and something else be used like HashiCorp Vault?

Assaf: HashiCorp is just one solution, but solutions also bring problems, like who controls the mega secret of HashiCorp and what happens if that gets breached. Also, the complexities because sometimes it... brings new complexities and new problems to implement, which could affect the business and the pace of production and working. So with each solution, we have a new set of security and operational problems. I guess secrets have their purpose. They have their purpose, and we need them. HashiCorp is one good example of implementation. There are other good examples as well. People should understand their security appetite compared to their operational needs and use that. But I wouldn't say to completely abandon secrets, just be aware and understand the implications, specifically to Kubernetes clusters.

Bart: I mentioned earlier in the conversation that we had an episode about this with one of our guests named Mac, and he shared that Kubernetes secrets are mostly fine. He arrived at this conclusion by building his own threat model and arguing that most other solutions don't offer a significant jump in security. What is your advice regarding the perimeter of your cluster?

Assaf: So I actually saw that episode and read the article. First of all, kudos. I really loved it. I'm 100% excited to see that the engineers are keen on security, which is a good step. Yakir said that there are not a lot of people. But I think he's missing a few points. It's not enough what it says. It's missing, for instance, supply chain attacks, which are not taken into consideration. And I think also the etcd, what happens post-exploitation? He just talked about the initial access, which is one point in the attack kill chain vector. But what happens if someone accesses your cluster and reads the etcd or reads files from the node, which is another problem. I strongly advise adopting a more mature Kubernetes threat model. You have a threat model by Microsoft, for instance. They have a very good Kubernetes threat model, which is updated. There you can see not only the initial access, but the lateral movement and also exfiltration. which is a more mature security way of thought, if that makes sense.

Bart: When we think about security, Kubernetes and the cloud-native world are already rather niche. Getting into the security space is even more so. But for the two of you, how did your passion for security start? Did you, as a 10-year-old, say, "When I'm older, I want to be a cybersecurity expert"? Maybe you saw a movie. What was it that got the path started for you?

Yakir: Actually, I was a developer in my past. I was a full stack developer and I enjoyed it. I remember that one time I did capture the flag. This kind of event requires you to break something or exploit a specific vulnerability. This was the first time I was exposed to the world of cybersecurity and how huge it is. After that, I never looked back. I found a new passion, and I think this gave me real meaning for my career because the things you do can really impact a lot of people in the industry. I realized that I knew very little and needed to expand my knowledge in different topics. There are many different concepts waiting for discovery. For example, if you look at the CNCF landscape or the full stack technology, imagine that for each one of them, there might be, as a security researcher, an interesting vulnerability or attack vector that the organization isn't familiar with yet. There are many things to learn and discover in this field.

Bart: What about you, Assaf? How did your career in cybersecurity start?

Assaf: So I got my first computer when I was six years old. With no instruction, I started to break things and tried to circumvent games, hack into them, and bypass things. This is how things started for me. I actually abandoned it for a few years and then came back when I was in the... high-tech industry. This passion returned to me, and ever since, I've been trying to define new boundaries of security, better understand security posture, and read everything. I share the passion with Yakir to read everything, to understand everything.

Yakir: If it's not documented, we must do it.

Assaf: If it's not documented, then... probably we can find it beyond the documentation. And if it's documented, it's just sitting in plain sight. You need to make the connections. And as you see, Yakir said, we all need to do that. We need to read all the time to see tutorials, better understand things.

Bart: Yakir, do you want to speak about the importance of reading?

Yakir: Reading of cyber materials or security materials?

Bart: No, poetry, romantic novels.

Yakir: So when I read some cookbooks... I think as a security researcher, you must first think as DevOps or the developer of the technology in order to really understand the basics and what they do when they use this technology. After this, you will understand the new attack vector or a new risk. These are the things that attackers do as well when they try to hunt vulnerabilities. For example, when there is a release note of a new version of Kubernetes or other software... They try to find if there is any fix for a vulnerability that maybe the company hasn't disclosed responsibly and exploit previous versions of this vulnerability. I think that we all must do this and stay up to date. We do not have another choice. Maybe watch a video after a conference. We can see it at KubeCon. In the last KubeCon, there were a lot of security talks about Kubernetes. The awareness is increasing.

Bart: Cloud Native Security Con has grown a lot as well, apart from KubeCon. That's why I said previously that it seems everyone nowadays, in one way or another, is involved in security. It's something that can't just be isolated inside a team. I really like the point you mentioned about finding your local security researcher. Don't be ashamed to ask for help or say, "Hey, I don't know what are some of the best practices." GitHub, for example, has developed features that some people may not be aware of, which are accessible. It's not just, "Well, I'm gonna wait for this to be a problem, then someone's gonna clean it up." No, you can be part of the solution. Also, understanding that as security professionals, you can make an impact for other people. Yakir, before you started at Aqua, you were in a red team before switching to a blue team. This sounds a little bit like the Matrix. What is a red team? What is a blue team?

Yakir: I think that Assaf will add more information about the blue sides, even though you have some red attributes. The Red Teamer is actually a security role within an organization. They attempt to simulate adversary behavior in the network and assess the organization's response to cybersecurity incidents. As part of this role, we sometimes develop fake malware or conduct campaigns to check the security of the company network and to prove to the company how long it took them to detect suspicious activity and what the attacker was able to do in this time. Actually, I have never switched sides. I am on the red side currently. I try to uncover vulnerabilities. When we say red side, we mean more offensive research. We try to find vulnerabilities, attack vectors, and risks, and we prefer this over something called reverse engineering or finding the attackers.

Assaf: I think red and blue come from the army, from war games. The reds are the attackers and the blue are the defenders. The defenders need to search their own perimeter to understand if there is a breach, what attacked them, to collect the IOCs or indications of compromise, indications of attacks, to understand if there is an active attack or if there was an attack in the past, and also to understand if the security borders, the controls, the perimeter is not breached, to defend it, to understand how to defend it. It's the attackers and defenders, and Yakir mentioned something in between. There is another third option, which is purple, combining both attacker and defender roles. This mindset involves thinking about how to defend things and how to attack. I think we lean more towards the purple side, or today, Yakir is more on the purple side. Even though he has more offensive thinking, he always considers how to defend, how to fix, how to add things to the current products or whatever we have to mitigate or defend others.

Bart: From a mentality perspective, that honestly sounds a little bit tiring, attacking yourself and defending yourself at the same time. But if you're able to do it, I respect you. Now for the two of you, what's next? You give a lot of talks and are very active in the community. What's next? What can we expect from you in the coming six months to a year?

Yakir: Actually, we are currently conducting a significant research study on a lot of undetectable vulnerabilities. on GitHub and GitLab, and there are no scanning tools available nowadays. So, I think it will create some noise when it is released in the cloud-native security community. Currently, we are examining different cloud vulnerabilities. We'll speak at Black Hat about various vulnerabilities we found on AWS that allow us to breach any AWS account in the world.

Assaf: Assume total control over AWS accounts.

Yakir: We reported this finding to AWS, which fixed it, and we will release this at Black Hat.

Assaf: Over the past couple of years, we took Kubernetes to the next level. We were trying to create Kubernetes honeypots in many areas like Kubelet and etcd. We are going to publish some reports about that. In order to create a honeypot on the infrastructure of Kubernetes, we are also trying to find new things that others didn't find. We are also looking into that. So I guess more on the same on the equivalent.

Bart: Now that we talked about this earlier, we want to make security more accessible to developers and stakeholders who may not be aware of these concerns. What's the best way for people to get in touch?

Assaf: We have a blog, so if they want to get in touch with us, they can read our blogs. We have many blogs on Kubernetes, Cloud Native, and other areas. They can also reach out to me via LinkedIn or Twitter.

Yakir: Same for me. We would love to follow you.

Bart: You were easy to find. You weren't very secret. You were easy to find. You're busy, but not easy to find. Well, thank you very much for joining us today on KubeFM. Really enjoyed hearing from you about the work that you're doing. I think that there just needs to be the normalization of security as an everyday living, breathing aspect of every organization, regardless of being enterprise startup, it doesn't matter. It's something that can't be overlooked. And it doesn't have to be a culture of paranoia. It has to be a culture of proactivity and asking questions, finding the best resources and speaking to experts like the two of you. Thank you very much for your time today and look forward to seeing you somewhere in the Kubernetes universe.

Assaf: Thank you very much.

Bart: All right. Take care.