Skip to content

kubernetes

Fixing Karpenter Bottlerocket maxPods with VPC CNI Prefix Delegation

Karpenter

When provisioning worker nodes using Karpenter with AWS Bottlerocket OS and VPC CNI prefix delegation (ENABLE_PREFIX_DELEGATION=true), you may notice that node capacity is restricted to standard limits (e.g. 29 pods on m6i.large) rather than the expected 110 pods.

Root Causes & Fixes

1. Require Nitro-Based EC2 Instances

VPC CNI prefix delegation is exclusively supported on AWS Nitro hypervisors (c5, m5, m6i, c7g, etc.). Older Xen-based instance families do not support prefix delegation.

Enforce Nitro instances in your Karpenter NodePool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: primary-nodepool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-hypervisor"
          operator: In
          values: ["nitro"]

2. Override Kubelet maxPods in EC2NodeClass

When Karpenter initializes Bottlerocket nodes, Kubelet calculates max pods based on ENI counts unless explicitly overridden in the EC2NodeClass specification.

In Karpenter v1, configure spec.kubelet.maxPods:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: bottlerocket-custom
spec:
  amiFamily: Bottlerocket
  role: "KarpenterNodeRole-demo"
  kubelet:
    maxPods: 110
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "eks-demo"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "eks-demo"

How to Reset the ArgoCD Admin Password

ArgoCD

ArgoCD initial administrator credentials are automatically generated during installation and stored in the argocd-initial-admin-secret Kubernetes Secret.

Once customized or deleted, the active admin password hash is maintained inside the argocd-secret Secret. If the admin password is lost or locked out, follow the recovery procedure below.

Password Reset Workflow

graph LR
    Step1["1. Patch argocd-cm ConfigMap<br/>(Enable admin account)"] --> Step2["2. Remove admin.password key<br/>from argocd-secret"]
    Step2 --> Step3["3. Restart argocd-server Pod<br/>(Regenerates default password)"]

Step 1. Ensure the Admin Account is Enabled

Ensure admin.enabled: true is present in the argocd-cm ConfigMap:

kubectl patch -n argocd configmap argocd-cm --type merge -p '{"data":{"admin.enabled":"true"}}'

Step 2. Remove Existing Password Hash from argocd-secret

Remove the admin.password and admin.passwordMtime keys from the argocd-secret Secret:

kubectl patch secret argocd-secret -n argocd --type json \
  -p='[{"op": "remove", "path": "/data/admin.password"}, {"op": "remove", "path": "/data/admin.passwordMtime"}]'

Example structure of argocd-secret:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
data:
  server.secretkey: K1ZCZlpEeWYwMFpjUzV5NG5tTUROOFllS0plYz0=

Step 3. Restart the ArgoCD Server Pod

Restarting the argocd-server deployment causes it to detect the missing admin password hash and regenerate the initial admin password:

kubectl rollout restart deployment argocd-server -n argocd

# Retrieve newly generated password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo

Complete Amazon EKS Cluster Build with Terraform & ALB

EKS Architecture

Creating a functional Amazon EKS cluster from scratch requires configuring VPC subnet discovery tags, managed node groups, VPC CNI prefix delegation, and the AWS Load Balancer Controller.

📁 Source Code: aws-eks-terraform / EKS-Cluster-ALB

📺 Video Walkthrough

What's Included

  • VPC: 2 private subnets for worker nodes and 2 public subnets with ELB discovery tags.
  • EKS Cluster: Kubernetes 1.31+ with API-only access authentication (EKS Access Entries).
  • Managed Node Group: Single Spot instance running optimized Bottlerocket OS.
  • VPC-CNI Add-on: Enabled with prefix delegation for high pod density.
  • AWS Load Balancer Controller: Deployed via Helm to provision ALBs in ip target mode.

Mounting AWS Secrets as Environment Variables via Secrets Store CSI Driver

Kubernetes CSI

By default, the AWS Secrets Store CSI Driver mounts secrets from AWS Secrets Manager as volume-attached files inside containers.

However, many legacy and cloud-native applications expect secrets (such as database credentials or API tokens) to be passed as environment variables.

By utilizing spec.secretObjects in the SecretProviderClass, the CSI driver automatically creates a synchronized native Kubernetes Secret whenever a pod mounts the volume, allowing you to consume the secrets as standard env variables.

📁 Source Code: Aws-Eks-SecretsManager

SecretProviderClass Configuration

secret-provider-class.yaml 

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secret-to-k8s-secret
  namespace: default
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "MySecret"
        objectType: "secretsmanager"
        jmesPath:
          - path: "username"
            objectAlias: "DB_USER"
          - path: "password"
            objectAlias: "DB_PASS"
  secretObjects:
    - secretName: db-k8s-secret
      type: Opaque
      data:
        - objectName: "DB_USER"
          key: "username"
        - objectName: "DB_PASS"
          key: "password"

Pod Manifest Mounting Secret as Environment Variable

pod.yaml 

apiVersion: v1
kind: Pod
metadata:
  name: demo-app
  namespace: default
spec:
  serviceAccountName: nginx-deployment-sa
  containers:
    - name: app
      image: nginx:alpine
      env:
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: db-k8s-secret
              key: username
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-k8s-secret
              key: password
      volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secret-to-k8s-secret"

Volume Mount Required

The Secrets Store CSI Driver creates the Kubernetes Secret only when the volume is actively mounted by at least one running Pod.

EKS: Avoiding 502/504 Errors & Timeouts During Rolling Deployments (ALB)

Target Group Status

When exposing Amazon EKS applications through an AWS Application Load Balancer (ALB) using IP target mode, rolling updates can sometimes trigger brief 502 Bad Gateway / 504 Gateway Timeout errors.

Root Causes

  1. Slow Target Registration: Kubernetes marks a new pod as Ready before the AWS ALB Target Group finishes health checking and registering the pod's IP address.
  2. Abrupt Termination: Kubernetes sends SIGTERM and terminates an old pod before the ALB finishes draining active connections from the target group.

The Two-Part Solution

1. Enable Pod Readiness Gates

Configuring Pod Readiness Gates prevents Kubernetes from terminating old pods until the new pod's IP is fully registered and healthy inside the AWS ALB target group.

Enable readiness gate injection by labeling your application namespace:

kubectl label namespace default elbv2.k8s.aws/pod-readiness-gate-inject=enabled

2. Configure preStop Sleep Hook & Deregistration Delay

Add a preStop hook to delay pod shutdown while the ALB deregisters the target, and tune the Target Group deregistration delay:

deployment.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: nginx:latest
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 30"]

ingress.yaml 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-service
                port:
                  number: 80