Skip to content

EKS Deployment & Architecture Planning Guide

EKS Architecture

Designing an enterprise-ready Amazon EKS cluster requires evaluating foundational architecture decisions across networking, compute, security, storage, ingress, and operational governance before writing Infrastructure as Code (Terraform).

This guide provides a structured planning checklist and architectural best practices for deploying production EKS environments.

1. VPC & Networking Topology

graph TD
    VPC["VPC (e.g. 10.0.0.0/16)"]
    PubSub["Public Subnets (ALB / Ingress / NAT GW)"]
    PrivSub["Private Subnets (EKS Worker Nodes)"]
    SecSub["Secondary Pod Subnets (e.g. 100.64.0.0/16)"]

    VPC --> PubSub
    VPC --> PrivSub
    VPC --> SecSub
Decision Area Architectural Considerations Recommendation / Best Practice
VPC CIDR & IP Strategy Pod IP exhaustion in default /16 or /20 subnets. Use VPC-CNI Custom Networking with a non-routable secondary CIDR (e.g. 100.64.0.0/16 or RFC 6598) for pods, preserving routable corporate IPs for nodes.
Subnet Allocation Redundancy across Availability Zones. Spread across at least 3 Availability Zones with dedicated private subnets for worker nodes and public subnets for external load balancers.
Subnet Tagging Required for AWS Load Balancer Controller discovery. Add kubernetes.io/role/elb = 1 on public subnets and kubernetes.io/role/internal-elb = 1 on private subnets.
NAT Gateways Outbound internet connectivity for private nodes. Provision 1 NAT Gateway per AZ for high-availability production workloads; use a single NAT Gateway for cost-sensitive dev/test environments.
VPC Endpoints (PrivateLink) Eliminates NAT Gateway data transfer costs for AWS services. Provision Interface Endpoints for ecr.api, ecr.dkr, s3 (Gateway), sts, logs, and ec2.
EKS Control Plane Endpoints Public vs. Private API endpoint access. Enable Private Access + Public Access (with CIDR allowlisting) or fully private with VPN/Transit Gateway/Bastion.

2. Cluster Control Plane & Access Control

Decision Area Architectural Considerations Recommendation / Best Practice
Cluster Mode Standard EKS vs. EKS Auto Mode. EKS Auto Mode for fully automated node provisioning, storage, and networking; Standard EKS with Karpenter v1 for fine-grained node customization and custom AMIs.
Authentication Mode Legacy aws-auth ConfigMap vs. EKS Access Entries. Use authentication_mode = "API" with EKS Access Entries. Avoid aws-auth ConfigMap on all new clusters (deprecated).
Kubernetes Version Cluster lifecycle and support windows. Deploy current stable version (e.g. 1.31+) and plan quarterly minor version upgrades.
Cluster Encryption Kubernetes Secret encryption at rest. Enable AWS KMS envelope encryption for cluster secrets (secrets resource type) with customer-managed KMS keys.

3. Compute & Autoscaling Strategy

Decision Area Architectural Considerations Recommendation / Best Practice
Autoscaling Engine Karpenter v1 vs. Cluster Autoscaler. Standardize on Karpenter v1 for sub-minute node provisioning, right-sizing, automatic bin-packing, and native Spot consolidation.
OS / AMI Family Bottlerocket vs. Amazon Linux 2023 (AL2023). Bottlerocket for minimal attack surface, atomic updates, and security hardening; AL2023 for standard Linux enterprise tooling.
Purchasing Strategy Spot vs. On-Demand vs. Savings Plans / Reserved Instances. Use Spot instances for stateless and batch workloads; use On-Demand / Reserved for stateful databases, controllers, and core system components.
Node Architecture x86 (AMD64) vs. ARM64 (Graviton). Build multi-arch images (arm64/amd64) to leverage AWS Graviton (c7g/m7g/r7g) for up to 40% better price-performance.
Hypervisor Requirement Nitro-based vs. Legacy Xen instances. Require Nitro-based instances (karpenter.k8s.aws/instance-hypervisor: nitro) to support VPC CNI prefix delegation and Fast ENI attachment.

4. Identity & Access Management (Pod Security)

sequenceDiagram
    participant Pod as Pod (Application)
    participant Agent as EKS Pod Identity Agent
    participant EKSAuth as EKS Auth Service
    participant STS as AWS STS

    Pod->>Agent: Request temporary AWS credentials
    Agent->>EKSAuth: Validate ServiceAccount + Pod ID
    EKSAuth->>STS: AssumeRole for pods.eks.amazonaws.com
    STS-->>Pod: Temporary AWS credentials returned
  • EKS Pod Identity (Preferred): Map Kubernetes ServiceAccounts directly to IAM roles without OIDC providers or ServiceAccount annotations. Works across clusters and simplifies IAM role reuse.
  • IAM Roles for Service Accounts (IRSA): Retain only when legacy tools or cross-account OIDC federations strictly require it.
  • Principle of Least Privilege: Never grant AdministratorAccess. Use granular IAM policies scoped to specific S3 buckets, DynamoDB tables, or KMS keys.

5. Ingress & Traffic Management

Option Best Used For Key Advantages
Kubernetes Gateway API + Envoy Gateway Modern North/South & East/West traffic routing. Standardized API, traffic splitting (canary), Header matching, IP-mode NLB integration, direct pod delivery without ALB delays.
AWS Load Balancer Controller (ALB) Native AWS Application Load Balancer integration. AWS WAF integration, ACM SSL termination, Cognito authentication, path/host routing.
Ingress-NGINX In-cluster reverse proxy with complex rewrite rules. Advanced NGINX annotations, internal ClusterIP routing, Prometheus metrics, Cert-Manager integration.

Prevent 502/504 Errors on Deployments

When using AWS Load Balancer target type ip, always configure Pod Readiness Gates (targetgroupbinding.elbv2.k8s.aws), preStop sleep hooks (15-30s), and adjust target group deregistration delays (30s) to avoid traffic dropping during rolling updates.

6. Stateful Storage & State Management

  • Amazon EBS (EBS CSI Driver): High-performance single-AZ block storage (gp3). Enforce encryption via custom StorageClasses with customer-managed KMS keys.
  • Amazon EFS (EFS CSI Driver): Multi-AZ shared file systems for read-write-many (ReadWriteMany) workloads.
  • Amazon S3 Files: Direct NFS-compatible, cached access to S3 data lake buckets using EFS CSI Driver v3.0+ and Pod Identity.

7. Security, Governance & Secrets

  • Secrets Management: Use the AWS Secrets Manager / SSM Parameter Store via External Secrets Operator (ESO) or the Secrets Store CSI Driver with auto-rotation.
  • Network Policies: Enforce zero-trust pod-to-pod network segmentation using VPC CNI Network Policy Engine (enabled via enableNetworkPolicy: "true").
  • Container Security: Scan container images in ECR with AWS Inspector; enforce admission control via Kyverno or OPA Gatekeeper.

8. Observability & GitOps CI/CD

  • Metrics & Logging: Prometheus + Grafana for cluster metrics; Fluent Bit sending application and audit logs to CloudWatch or OpenSearch.
  • Continuous Delivery: ArgoCD or Flux for declarative, version-controlled cluster state management (GitOps).
  • Automated DNS & TLS: ExternalDNS (Route53 sync) + Cert-Manager (Let's Encrypt / AWS Private CA).