EKS Auto and Karpenter with VPC Secondary IP
AWS VPC supports secondary CIDR ranges that can be assigned to EKS pods. This means you can have a smaller, routable CIDR range for nodes while using a larger secondary IP range for pod networking.
To provision pods in a secondary IP range with EKS Auto Mode, the following is required:
- Install/update the VPC-CNI add-on with custom networking and prefix delegation enabled
- Create an ENIConfig for each pod subnet in each availability zone
- Create a custom Karpenter NodeClass and NodePool with a higher priority than the default NodePool
Source code:
Complete Setup
In this example, I create a VPC with a primary CIDR of 10.0.0.0/24 spread across 3 AZs and add a secondary CIDR of 100.0.0.0/16 for pods.
Applying the code:
terraform init -upgrade
terraform apply
# Once EKS is ready, update kubeconfig
aws eks --profile labs --region eu-west-1 update-kubeconfig --name eks-auto-demo
kubectl cluster-info
# Apply custom NodePool
kubectl apply -f Bootstrap/cni-nodepool.yaml
# Deploy a pod and verify it receives an IP from the secondary range
kubectl run nginx --image=nginx
Step 1. Create VPC
Create a VPC with the necessary CIDR range. The VPC module supports secondary IPs, but here I create it separately so custom tags can be applied.
Step 2. Define the secondary IP range
# Create secondary CIDR once VPC is created
resource "aws_vpc_ipv4_cidr_block_association" "secondary_cidr" {
vpc_id = module.vpc.vpc_id
cidr_block = local.secondary_cidr_block
}
resource "aws_subnet" "podnet" {
for_each = {
for idx, az in local.azs :
az => {
cidr_block = local.pod_subnets[idx]
}
}
vpc_id = module.vpc.vpc_id
availability_zone = each.key
cidr_block = each.value.cidr_block
tags = {
"Name" = "${module.vpc.name}-pod-subnet${each.key}"
"kubernetes.io/role/internal-elb" = "1"
"kubernetes.io/role/cni" = "1"
"pod_subnet" = "true"
"subnet_purpose" = "EKS_Cluster"
}
depends_on = [aws_vpc_ipv4_cidr_block_association.secondary_cidr]
}
resource "aws_route_table_association" "podnet-association" {
for_each = aws_subnet.podnet
subnet_id = each.value.id
route_table_id = module.vpc.private_route_table_ids.0
}
Step 3. Create the EKS Auto cluster
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "21.0.6"
name = "eks-auto-demo"
kubernetes_version = "1.33"
endpoint_public_access = true
enable_cluster_creator_admin_permissions = true
compute_config = {
enabled = true
node_pools = ["general-purpose"]
}
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
tags = {
Environment = "dev"
Terraform = "true"
Owner = "D Vettom"
}
}
Step 4. Install VPC-CNI and create ENIConfigs per pod subnet
Enable custom networking and prefix delegation, and create an ENIConfig for each pod subnet/AZ.
locals {
az = [for k, v in aws_subnet.podnet : v.availability_zone]
subnet_id = [for k, v in aws_subnet.podnet : v.id]
}
resource "helm_release" "vpc-cni" {
name = "aws-vpc-cni"
namespace = "kube-system"
repository = "https://aws.github.io/eks-charts"
chart = "aws-vpc-cni"
version = "1.19.6"
values = [
<<-EOT
env:
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: "true"
ENABLE_POD_ENI: "true"
ENABLE_PREFIX_DELEGATION: "true"
ENABLE_SUBNET_DISCOVERY: "true"
eniConfig:
create: true
region: "${data.aws_region.current.region}"
subnets:
"${local.az[0]}":
id: "${local.subnet_id[0]}"
securityGroups:
- "${module.eks.node_security_group_id}"
"${local.az[1]}":
id: "${local.subnet_id[1]}"
securityGroups:
- "${module.eks.node_security_group_id}"
"${local.az[2]}":
id: "${local.subnet_id[2]}"
securityGroups:
- "${module.eks.node_security_group_id}"
EOT
]
}
Step 5. (Optional) Create a security group for pod networking
You can use the cluster's existing security group or create a dedicated one for pods on the secondary CIDR.
resource "aws_security_group" "pods_sg" {
name = "eks-auto-secondary-pods-sg"
description = "Security group for EKS Auto pods using secondary IP addresses"
vpc_id = module.vpc.vpc_id
tags = {
Name = "eks-auto-secondary-pods-sg"
pod_security_group = "true"
purpose = "SG for pods using secondary IP address"
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["10.0.0.0/8", "100.0.0.0/8"]
}
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["10.0.0.0/8", "100.0.0.0/8"]
}
ingress {
from_port = 1025
to_port = 65535
protocol = "tcp"
cidr_blocks = ["100.0.0.0/8"]
}
ingress {
from_port = 53
to_port = 53
protocol = "udp"
cidr_blocks = ["100.0.0.0/8"]
}
}
resource "aws_vpc_security_group_egress_rule" "allow_all_traffic_ipv4" {
security_group_id = aws_security_group.pods_sg.id
description = "Allow all outbound traffic"
cidr_ipv4 = "0.0.0.0/0"
ip_protocol = "-1"
}
Step 6. Apply a custom Karpenter NodePool
Create a custom NodeClass with securityGroupSelectorTerms and podSecurityGroupSelectorTerms. Update the NodeClass with podSecurityGroupSelectorTerms and podSubnetSelectorTerms.
The NodePool must reference the custom NodeClass and have a weight greater than 0 to take precedence over the EKS Auto default NodePool.
The NodeClass must include securityGroupSelectorTerms, podSecurityGroupSelectorTerms, subnetSelectorTerms, and podSubnetSelectorTerms. Ensure every node subnet has a corresponding pod subnet with an ENIConfig.
[!NOTE] Configure the NodePool to provision Nitro instances — Xen does not support CNI prefix delegation.
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
finalizers:
- eks.amazonaws.com/termination
name: primary-nodeclass
spec:
ephemeralStorage:
iops: 3000
size: 80Gi
throughput: 125
networkPolicy: DefaultAllow
networkPolicyEventLogs: Disabled
role: AmazonEKSAutoNodeRole
securityGroupSelectorTerms:
- tags:
"Name": "eks-auto-demo-node"
podSecurityGroupSelectorTerms:
- tags:
"Name": "eks-auto-demo-node"
snatPolicy: Random
subnetSelectorTerms:
- tags:
subnet_type: "private"
subnet_purpose: "EKS_Cluster"
podSubnetSelectorTerms:
- tags:
pod_subnet: "true"
subnet_purpose: "EKS_Cluster"
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: primary-nodepool
spec:
weight: 20
template:
spec:
expireAfter: 336h
nodeClassRef:
group: eks.amazonaws.com
kind: NodeClass
name: primary-nodeclass
requirements:
- key: "karpenter.k8s.aws/instance-hypervisor"
operator: In
values: ["nitro"]
- key: karpenter.sh/capacity-type
operator: In
values: ["spot","on-demand"]
- key: eks.amazonaws.com/instance-category
operator: In
values: ["c","m","r"]
- key: eks.amazonaws.com/instance-generation
operator: Gt
values: ["5"]
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: kubernetes.io/os
operator: In
values:
- linux
terminationGracePeriod: 24h0m0s
Test pod IPs
Deploy a pod and verify that the node IP is from the private subnet and the pod IP is from the secondary CIDR range.