Container ecosystems: From runtime fundamentals to purpose-built Kubernetes

Bart: I'm Phil Estes. I work for Amazon Web Services (AWS).

Phil: I'm Phil Estes, a principal engineer at AWS. I've been there about four years and focus on upstream open source in the container runtime space. I was involved in Docker for a long time and am a containerd maintainer, a CNCF graduated project. I help lead some open source activity around that area and the OCI from Amazon.

Bart: I notice the transcript snippet is incomplete. Could you provide the full context of Phil Estes' response about emerging Kubernetes tools?

Phil: The exciting part about joining Amazon four years ago is seeing their growth in open source contribution and project creation. Karpenter and KRO are two projects that came out of Amazon and are now used by other cloud providers. These are exciting pieces for creating and managing clusters and handling the complexity of resource management. Additionally, a fellow principal engineer created K8sPGT, which brings the power of large language models to diagnosing cluster issues by analyzing error messages. It's now a CNCF sandbox project. These three projects all begin with K and are really impressive.

Bart: One of our podcast guests, Emin, pointed out that Go processes and containers aren't aware of resource limits. Have you encountered any unexpected interactions between programming languages and Kubernetes resource management?

Phil: There has been a long-term issue with languages that have a virtual machine, like Java with the JVM, which tries to manage its own heap memory. People tend to start Java processes with specific limits and then put them in a container with its own cgroups controls around resources.

From my involvement in Docker and ContainerD in the early days, people have been trying to resolve differences about who's managing memory usage. Users are solving these challenges and can find information online about matching awareness of memory capacity and actual memory use. People must essentially talk to Linux to understand the real state of their memory usage, which is how container runtimes handle it using cgroups.

Regarding the statement "Kubernetes is just Linux," the basic idea is that Linux containers are wrapping Linux processes, and everything within a container is a Linux process using libraries and kernel interfaces. This means users must understand enough about Linux to comprehend why their container works or doesn't work in certain ways.

There's an expectation that users need awareness of how to package containers with the right components and libraries, similar to deploying on a Linux host. Many of these principles directly translate into the Kubernetes and container ecosystem.

Bart: Harsha explained that container security doesn't stop at using minimal container images. Instead, one should look at the entire supply chain, including dependencies, software composition analysis, and software build materials (SBOM).

Phil: There are tons of experts in this space and vendors focusing on secure supply chain capabilities and tools. One interesting piece that Amazon is investing in, to help the community, is recognizing that most dependencies come from package managers like Node.js or Go. Historically, we've downloaded bits from the internet without always being able to verify their origin.

The focus is on image signing, package signing, and certificate management. The Open SSF has even initiated two-factor authentication efforts, providing maintainers with physical keys to properly sign and verify packages. This approach is crucial to move away from simply downloading internet bits and hoping they're secure.

We've seen too many security exploits involving package subversion and typosquatting. Having a model of signature verification across the entire stack is key to achieving a secure supply chain.

Bart: Mircea argued that you shouldn't use a Linux distribution like Ubuntu to run your Kubernetes nodes and control plane. If your goal is to host Kubernetes, you should use policies and consider minimal distributions like Talos and BottleRocket. What's your advice on operating systems in Kubernetes?

Phil: One of my sister teams at AWS is the BottleRocket OS team, which is a purpose-optimized Linux that's secure and purpose-built for running containers. Even if you're not into BottleRocket or Talos, the concept is that whatever OS you pick should be minimal because all you're trying to do is run containers. You need some basic elements to run containers and have logging and monitoring capabilities on your node. Beyond that, you don't want a big, heavy host OS that you'll have to patch and update.

These purpose-optimized OSs are definitely the right path because you have to think about more than just the operating system. You must consider how you're getting Kubernetes and a container runtime. The fact that these systems are packaging, maintaining, and patching these pieces for you is a significant step in management automation for your host nodes in Kubernetes.

Bart: Kubernetes turned 10 years old last year. What should we expect in the next 10 years?

Phil: As we look around, the core of Kubernetes is maturing, but there's still a ton of innovation. We're seeing new challenges like running LLMs in Kubernetes or handling complex networking stacks. The core is getting mature, and as new technologies emerge, the Kubernetes community is exploring how Kubernetes works with serverless, AI, and other emerging technologies. I think this is the kind of evolution we'll continue to see in the next 10 years.

Bart: The transcript snippet is very short and doesn't provide much context about what might be "next" for Phil. Without more context from the surrounding conversation, I cannot confidently add hyperlinks. Could you provide more of the transcript or context about the conversation?

Phil: What's next for me? I've been stuck in this container runtime world for a decade, and I love it. I'm going to keep plugging away. The cool thing is, I gave a talk Monday this week on maintaining the maintainers—how our projects age and how we'll bring along new talent. I have gray hair, so I'm not going to be working forever. I'm doing a lot of mentoring internally at Amazon and in our communities. I'd love to see myself not writing so much code, but helping others come up, adopt open source methodologies, and get involved in projects. That's where I see myself.

Bart: How can people get in touch with you?

Note: While the transcript doesn't provide specific context about contact details, based on the context of the conversation and the links available, I would suggest adding a link to the Kubernetes Slack as a potential way for people to connect with Phil Estes.

Phil: Luckily, I've tried to get the same name everywhere. So Estes and my first initial P. You can find me on GitHub, Blue Sky, LinkedIn, CNCF, and Kubernetes Slack. I'm always happy to answer questions or talk to folks who are interested in open source.