EKS Auto and Karpenter with VPC Secondary IP

In large enterprise networks, allocating thousands of routable IPv4 addresses to Kubernetes pods can quickly exhaust the corporate VPC IP space.

AWS VPC supports adding Secondary IPv4 CIDR blocks (e.g. 100.64.0.0/16 / RFC 6598 carrier-grade NAT or 100.x.x.x). By configuring VPC-CNI Custom Networking and custom Karpenter NodeClasses, worker nodes consume a small, routable primary CIDR while all pods receive IPs from the large secondary CIDR range.

📁 Source Code: aws-eks-terraform / EKS-Auto-secondary

---

📺 Video Walkthrough

---

Architecture Overview

graph TD
    subgraph VPC ["VPC (Primary CIDR: 10.0.0.0/24 & Secondary CIDR: 100.64.0.0/16)"]
        subgraph NodeSubnet ["Node Subnets (Primary: 10.0.0.x)"]
            Node["EKS Node (Primary ENI: 10.0.0.15)"]
        end
        subgraph PodSubnet ["Pod Subnets (Secondary: 100.64.x.x)"]
            SecondaryENI["Secondary ENI (100.64.1.20)"]
            PodA["Pod A (100.64.1.21)"]
            PodB["Pod B (100.64.1.22)"]
        end
    end
    Node --> SecondaryENI
    SecondaryENI --> PodA
    SecondaryENI --> PodB

---

Step-by-Step Implementation

Step 1. Define Secondary CIDR Block and Pod Subnets

Attach a secondary CIDR to the VPC and provision subnets across Availability Zones:

secondary-cidr.tf

resource "aws_vpc_ipv4_cidr_block_association" "secondary_cidr" {
  vpc_id     = module.vpc.vpc_id
  cidr_block = "100.64.0.0/16"
}

resource "aws_subnet" "podnet" {
  for_each = {
    for idx, az in local.azs :
    az => {
      cidr_block = local.pod_subnets[idx] # e.g. ["100.64.0.0/19", "100.64.32.0/19", "100.64.64.0/19"]
    }
  }

  vpc_id            = module.vpc.vpc_id
  availability_zone = each.key
  cidr_block        = each.value.cidr_block

  tags = {
    "Name"                            = "${module.vpc.name}-pod-subnet-${each.key}"
    "kubernetes.io/role/internal-elb" = "1"
    "kubernetes.io/role/cni"          = "1"
    "pod_subnet"                      = "true"
    "subnet_purpose"                  = "EKS_Cluster"
  }

  depends_on = [aws_vpc_ipv4_cidr_block_association.secondary_cidr]
}

resource "aws_route_table_association" "podnet_association" {
  for_each       = aws_subnet.podnet
  subnet_id      = each.value.id
  route_table_id = module.vpc.private_route_table_ids[0]
}

---

Step 2. Configure VPC-CNI with Custom Networking & ENIConfigs

Enable AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG in VPC-CNI and generate an ENIConfig resource for each Availability Zone:

cni.tf

resource "helm_release" "vpc_cni" {
  name       = "aws-vpc-cni"
  namespace  = "kube-system"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-vpc-cni"
  version    = "1.19.0"

  values = [
    <<-EOT
    env:
      AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: "true"
      ENABLE_POD_ENI: "true"
      ENABLE_PREFIX_DELEGATION: "true"
      ENABLE_SUBNET_DISCOVERY: "true"
    eniConfig:
      create: true
      region: "${data.aws_region.current.region}"
      subnets:
        "${local.azs[0]}":
          id: "${aws_subnet.podnet[local.azs[0]].id}"
          securityGroups:
            - "${module.eks.node_security_group_id}"
        "${local.azs[1]}":
          id: "${aws_subnet.podnet[local.azs[1]].id}"
          securityGroups:
            - "${module.eks.node_security_group_id}"
    EOT
  ]
}

---

Step 3. Create Karpenter NodeClass with Pod Subnet Selectors

Configure the NodeClass with podSubnetSelectorTerms to bind nodes to the secondary CIDR pod subnets:

custom-nodeclass-nodepool.yaml

apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: primary-nodeclass
spec:
  ephemeralStorage:
    iops: 3000
    size: 80Gi
    throughput: 125
  networkPolicy: DefaultAllow
  role: AmazonEKSAutoNodeRole
  securityGroupSelectorTerms:
    - tags:
        "Name": "eks-auto-demo-node"
  podSecurityGroupSelectorTerms:
    - tags:
        "Name": "eks-auto-demo-node"
  snatPolicy: Random
  subnetSelectorTerms:
    - tags:
        subnet_type: "private"
        subnet_purpose: "EKS_Cluster"
  podSubnetSelectorTerms:
    - tags:
        pod_subnet: "true"
        subnet_purpose: "EKS_Cluster"
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: primary-nodepool
spec:
  weight: 20
  template:
    spec:
      expireAfter: 336h
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: primary-nodeclass
      requirements:
        - key: "karpenter.k8s.aws/instance-hypervisor"
          operator: In
          values: ["nitro"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: eks.amazonaws.com/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: eks.amazonaws.com/instance-generation
          operator: Gt
          values: ["5"]
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64
        - key: kubernetes.io/os
          operator: In
          values:
            - linux
      terminationGracePeriod: 24h0m0s

---

Verification

# 1. Deploy test pod
kubectl run nginx --image=nginx

# 2. Check pod and node IP addresses
kubectl get pod nginx -o wide

Observe that the worker node's IP address belongs to the primary subnet CIDR (10.0.0.x), while the pod's IP address is assigned from the secondary subnet CIDR (100.64.x.x).