Skip to content

PREFIX_DELEGATION

Karpenter maxpods

Karpenter Nodes maxPods not working with Bottlerocket despite enabling VPC-Prefix

karpenter

Problem

- Bottle rocket nodes max pod capacity is less than it should be
- Karpenter provisioned bottlerocket does not respect VPC prefix deligation
- Max pods on a Large instance remain 29 despite enabling VPC prefix deligation
- Large instances only has capacity for 29 pods and not 110 with prefix

Solution

Assuming VPC previx is enabled with correct VPC CNI version, there are 2 main reasons for this. 1. Instance provisioned is not Nitro instance 2. Bottlerocket require max pods to be overriden

Explanation

VPC prefix works on AWS Nitro based hypervisor servers only. They comes with dedicated hardware for handling networking traffic and supports VPC Prefix delegation. OlderXen based server like M4,C4 etc does not have this support. So you must configure Karpenter to provision Nitro based instances only. This is defined in the Nodepool configuration

karpenter.k8s.aws/instance-hypervisor: nitro 

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: primary-nodepool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-hypervisor"
          operator: In
          values: ["nitro"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]

Second part of the problem is to do with Bottlerocket AMI. When Karpenter provisions the Node, it does not run script to check if Prefix delegation is enabled or not. This results in Max pods being calculated incorrectly. For example c6.large will show as supporting max 29 Pods.

Solution is to update Kubelet configuration to override default value. With Karpenter V1, Objects for setting Kubelet features have been moved from the NodePool spec to the EC2NodeClasses spec, to not require other Karpenter providers to support those features.

Update kubelet with maxPods: 110 

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: bottlerocket
spec:
  kubelet:
    maxPods: 110