Migrating from Amazon Linux to Bottlerocket while using Karpenter

We're starting a new series: Technical Nuggets on Platform Engineering: Beyond what you find in the documentation. We'll be sharing small nuggets of information on how Codurance's platform engineering team resolves some of the situations we find in our day-to-day, and we'll find that the answers often require going one step further than reading documentation. Let's start!

In this blog we're reviewing a new Karpenter update we did to our clusters that included a migration from Amazon Linux AMIs to Bottlerocket.

As Amazon defines on it’s website, 

“Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for running containers. Bottlerocket includes only the essential software required to run containers, and ensures that the underlying software is always secure.” - AWS Documentation

Our main goal is to achieve such a desired level of security and optimisation, that’s why we decided to migrate from Amazon Linux to Bottlerocket.

This configuration needs to be changed in Karpenter. Karpenter is an open-source node provisioning tool that automates cluster resizing based on workload demands, leveraging Kubernetes' native capabilities while offering support for cost-effective compute resources like spot instances. One of its main benefits is the easy way to configure these changes in the nodes.

By default the AMI configuration in Karpenter is AL2, so we now need to specify in our nodeClass the new AMI type:

Screenshot 2024-02-21 at 16.20.22

That would be everything we need to perform the change… if we didn't have any storage specifications!

In this case, we have some EBS specifications; for example we wanted to encrypt the volume and increase the disk size for our Kubernetes nodes:

Screenshot 2024-02-22 at 10.21.32

It’s important to take into account that Bottlerocket has two volumes:

  • An OS volume, used to store all OS related files and booting images
  • A data volume, used to store container data, container images, ephemeral volumes…

If we don’t change our current configuration, the OS volume will have an increased disk size and will be encrypted but the data volume will have the default configuration: reduced disk size and no encryption!

As the official Karpenter documentation states, we need to be aware of the special case and define the two blocks, configuring our data volume with the configuration we previously had for AL2:

Screenshot 2024-02-22 at 10.22.22

In case we have any additional doubts we can always find more info in the official Karpenter documentation.

Clicking on the banner you will learn about building world-class software