Karpenter Azure: A Comprehensive Setup and Configuration Guide

Karpenter Azure is a powerful tool that makes it easy to deploy and manage containerized applications on Azure. It's a highly scalable and flexible solution that can be used to create and manage clusters of any size.

To get started with Karpenter Azure, you'll need to create a Kubernetes cluster on Azure. This can be done using the Azure CLI or the Azure portal.

Karpenter Azure requires a Kubernetes cluster to function, so make sure you have one set up before proceeding. This will provide the foundation for your Karpenter Azure setup.

With a Kubernetes cluster in place, you can start configuring Karpenter Azure. This involves creating a Karpenter controller and a cluster configuration file.

To get started with Karpenter on Azure Kubernetes Service (AKS), you'll need to implement it into a demo AKS environment. We'll do so by installing Managed Karpenter, also known as Node Auto Provisioning.

Before diving in, ensure you have the following prerequisites in place: a demo AKS environment, and the necessary permissions to install Managed Karpenter. Be aware that there are specific limitations to consider.

The only network configuration allowed is Cilium + Overlay + Azure, so make sure your setup meets this requirement. Additionally, enabling this feature is not possible in clusters where node pools have the cluster autoscaler enabled.

You have the flexibility to use either a system-assigned or user-assigned managed identity, which is a nice feature. However, certain features are currently not supported, such as Windows node pools, applying custom configuration to the node kubelet, IPv6 clusters, and Service Principals.

To create a cluster, you'll need to create a new AKS cluster with the required configuration. This involves enabling workload identity and setting environment variables.

You can automate the process using a script, hack/deploy/create-cluster.sh, which takes in the cluster name, resource group, and namespace as arguments. This script will handle the creation of the AKS cluster and other necessary steps.

Here are the specific steps to create the AKS cluster:

To create a cluster, you can use a script called create-cluster.sh to automate the process. This script creates an AKS cluster with workload identity enabled, which is required for Karpenter.

The script sets environment variables, including the cluster name, resource group, and namespace. It also enables workload identity on the cluster.

To create a federated credential linked to the karpenter service account, you'll need to run a command that links the credential to the service account for authentication purposes.

Role assignments are also required to let Karpenter manage VMs and network resources. You can create these assignments by giving the MSI the roles "Virtual Machine Contributor", "Network Contributor", and "Managed Identity Operator" at the scope of the node resource group.

Here are the specific steps to give the MSI these roles:

To create a cluster, you need to create a pool. A single Karpenter NodePool can handle many different pod shapes. Karpenter makes scheduling and provisioning decisions based on pod attributes such as labels and affinity.

By creating a default NodePool, you can eliminate the need to manage many different node groups. This is done using the command below, which is also available in the repository under examples/v1beta1.

Karpenter is configured to reduce cluster cost by removing and replacing nodes when the consolidationPolicy is set to WhenUnderutilized. This means it will terminate any empty nodes on the cluster.

When configuring Karpenter for an AKS cluster, you'll need to specify certain values. The Karpenter Helm chart requires specific configuration values to work with an AKS cluster.

To generate the necessary configuration, you can use the configure-values.sh script. This script queries the AKS cluster and creates the karpenter-values.yaml file using karpenter-values-template.yaml as the template.

However, be aware that inconsistencies may arise between the installed version of Karpenter and the repository code. It's a good idea to download the specific version of the template before running the script.

To configure Helm chart values, you can use the configure-values.sh script which generates the karpenter-values.yaml file with necessary configuration.

This script queries the AKS cluster and creates karpenter-values.yaml using karpenter-values-template.yaml as the configuration template.

The script automatically fetches the template from the main branch, but inconsistencies may arise between the installed version of Karpenter and the repository code.

It's advisable to download the specific version of the template before running the script to avoid any potential issues.

The template is used to create the karpenter-values.yaml file, which is essential for the Karpenter Helm chart to work with an AKS cluster.

Weights are a crucial aspect of configuration options, allowing you to define the relative weight on your Node pool definitions.

This means you can set a preference of where a workload should be scheduled, giving you more control over resource allocation.

With multiple node pools defined, you can assign different weights to each, influencing the scheduling of workloads.

By doing so, you can ensure that critical workloads are prioritized and scheduled on the most suitable node pools.

Node pool weights enable you to fine-tune your resource utilization and optimize performance.

Before you start installing Karpenter on Azure Kubernetes Service (AKS), make sure you have the necessary prerequisites in place.

To get started, you'll need to install Managed Karpenter, also known as Node Auto Provisioning (Preview). This runs Karpenter as a managed addon similar to Managed Cluster Autoscaler.

Be aware that there are specific limitations to consider when using Node Auto Provisioning. The only network configuration allowed is Cilium + Overlay + Azure.

You can't enable this feature in clusters where node pools have the cluster autoscaler enabled. This is a key thing to keep in mind before proceeding.

Certain features are currently not supported, such as Windows node pools, applying custom configuration to the node kubelet, IPv6 clusters, and Service Principals.

Karpenter Azure offers two modes for Node Auto Provisioning: Node Auto Provisioning (NAP) mode and Self-hosted mode. NAP mode is the recommended mode for most users and is similar to managed Cluster Autoscaler.

In NAP mode, Karpenter is run by AKS as a managed addon. This mode is useful for users who want a hassle-free experience. You can follow the instructions in the Node Auto Provisioning documentation to use Karpenter in this mode.

Self-hosted mode, on the other hand, is useful for advanced users who want to customize or experiment with Karpenter's deployment. In this mode, Karpenter is run as a standalone deployment in the cluster. This mode requires more technical expertise and is not recommended for beginners.

You have two main options for using Karpenter with AKS: Node Auto Provisioning (NAP) mode and Self-hosted mode.

NAP mode is the recommended choice for most users, as it's managed by AKS and easy to set up.

Self-hosted mode, on the other hand, is designed for advanced users who want to customize or experiment with Karpenter's deployment.

To use NAP mode, you should follow the instructions in the Node Auto Provisioning documentation.

If you choose Self-hosted mode, you'll need to run Karpenter as a standalone deployment in the cluster.

Here are the two modes compared:

Karpenter can be used in two modes: Node Auto Provisioning (NAP) mode and Self-hosted mode. The recommended mode for most users is NAP mode.

To use Karpenter in NAP mode, you'll need to follow the instructions in the Node Auto Provisioning documentation.

Managed Karpenter, also known as Node Autoprovisioning, can be put to the test using Microsoft's Azure voting app. This demo app is a great way to see Karpenter in action.

To get started, you'll need to get the credentials for your AKS cluster and login interactively using the command `az login`.

After a few minutes, a new node will be spun up for the Azure Vote App workload to run on. This new node is managed by Karpenter.

You can use the command `kubectl get events -A --field-selector source=karpenter -w` to retrieve Kubernetes events across all namespaces and continuously watch for events where the source is specified as "karpenter".

These events provide insights into the context of Karpenter, indicating nominations on specific nodes and suggesting replacement with cheaper nodes is feasible.

Here are some useful resources to get you started with Karpenter:

You can manage node images in Azure Kubernetes Service (AKS) by setting the imageVersion on the node class. This allows you to pin a node pool at a certain node image version.

By default, NAP node pool virtual machines are automatically updated when a new image is available. If you want to revert to the latest node image version, simply remove the imageVersion spec from the node class definition.

You can set the imageVersion to one of the published releases listed on the AKS Release notes. For example, if you want to use the Ubuntu 22.04 image version "AKSUbuntu-2204-202311.07.0", you would set the imageVersion to "202311.07.0".

The imageVersion is the date portion on the Node Image, and only Ubuntu 22.04 is supported.