How to mount S3 Files on EKS

AWS S3 Files is a shared file system that connects any AWS compute directly with your data in Amazon S3. It presents S3 buckets as an NFS-compatible, cached EFS-backed filesystem that can be mounted on multiple nodes in a cluster — usable just like a standard EFS filesystem.

S3Files

In this example, I will document every step required to set up S3 Files and mount it in a pod using Terraform and Pod Identity. Complete code can be found in my GitRepo aws-eks-terraform/EKS-S3Files.

Prerequisites

EKS cluster with Pod Identity add-on enabled
VPC with private subnets for EKS nodes
Terraform with AWS provider configured

Install and configure EFS-CSI Driver

S3 Files are mounted on EKS using the EFS CSI driver. S3 Files presents a cached EFS volume in front of an S3 bucket, and mounting is handled by the EFS driver. EFS CSI driver version 3.0 or above is required for S3 Files support.

Step 1. Create IAM role for EFS CSI Driver

Create an IAM role and attach the EFS and S3 Files policies
Create and attach an explicit elasticfilesystem:ClientMount policy (required by Pod Identity)
Create Pod Identity associations for efs-csi-controller-sa and efs-csi-node-sa and attach the IAM role

iam-efs-csi-podid.tf

resource "aws_iam_role" "efs_csi_controller" {
  name               = "efs-csi-controller-${module.eks.cluster_name}"
  assume_role_policy = data.aws_iam_policy_document.pod_id_assume_role.json
}
resource "aws_iam_role_policy_attachment" "efs_csi_driver_policy" {
  role       = aws_iam_role.efs_csi_controller.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy"
}
resource "aws_iam_role_policy_attachment" "s3_files_csi_driver_policy" {
  role       = aws_iam_role.efs_csi_controller.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonS3FilesCSIDriverPolicy"
}
resource "aws_iam_role_policy_attachment" "efs_filesystem_utils" {
  role       = aws_iam_role.efs_csi_controller.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonElasticFileSystemsUtils"
}
resource "aws_iam_role_policy_attachment" "efs_csi_s3_client_full_access" {
  role       = aws_iam_role.efs_csi_controller.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3FilesClientFullAccess"
}
data "aws_iam_policy_document" "efs_client_mount" {
  statement {
    effect    = "Allow"
    actions   = ["elasticfilesystem:ClientMount"]
    resources = ["*"]
  }
}
resource "aws_iam_role_policy" "efs_client_mount" {
  name   = "efs-client-mount"
  role   = aws_iam_role.efs_csi_controller.name
  policy = data.aws_iam_policy_document.efs_client_mount.json
}
resource "aws_eks_pod_identity_association" "efs_csi_controller" {
  cluster_name    = module.eks.cluster_name
  namespace       = "kube-system"
  service_account = "efs-csi-controller-sa"
  role_arn        = aws_iam_role.efs_csi_controller.arn
}
resource "aws_eks_pod_identity_association" "efs_csi_node" {
  cluster_name    = module.eks.cluster_name
  namespace       = "kube-system"
  service_account = "efs-csi-node-sa"
  role_arn        = aws_iam_role.efs_csi_controller.arn
}

Step 2. Install EFS-CSI Driver

Available as a Helm chart at https://kubernetes-sigs.github.io/aws-efs-csi-driver/.

resource "helm_release" "aws_efs_csi_driver" {
  name             = "aws-efs-csi-driver"
  repository       = "https://kubernetes-sigs.github.io/aws-efs-csi-driver"
  chart            = "aws-efs-csi-driver"
  namespace        = "kube-system"
  create_namespace = false
  version          = "4.2.0"
  depends_on       = [module.eks]
}

S3 Files and S3 Bucket

The steps below cover everything required to create the S3 bucket, S3 Files filesystem, and access point.

Step 1. Create IAM role for S3 Files

When creating S3 Files via the console, an IAM role is assigned automatically. When using Terraform, you must create and attach the IAM role manually.

iam-s3-files.tf

data "aws_iam_policy_document" "s3_files_assume_role" {
  statement {
    sid     = "AllowS3FilesAssumeRole"
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["elasticfilesystem.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:s3files:${data.aws_region.current.region}:${data.aws_caller_identity.current.account_id}:file-system/*"]
    }
  }
}
resource "aws_iam_role" "s3_files" {
  name               = "s3-files-${module.eks.cluster_name}"
  assume_role_policy = data.aws_iam_policy_document.s3_files_assume_role.json
}
data "aws_iam_policy_document" "s3_files_bucket_access" {
  statement {
    sid    = "S3BucketPermissions"
    effect = "Allow"
    actions = [
      "s3:ListBucket",
      "s3:ListBucketVersions",
    ]
    resources = [aws_s3_bucket.s3filesstore.arn]
    condition {
      test     = "StringEquals"
      variable = "aws:ResourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
  statement {
    sid    = "S3ObjectPermissions"
    effect = "Allow"
    actions = [
      "s3:AbortMultipartUpload",
      "s3:DeleteObject*",
      "s3:GetObject*",
      "s3:List*",
      "s3:PutObject*",
    ]
    resources = ["${aws_s3_bucket.s3filesstore.arn}/*"]
    condition {
      test     = "StringEquals"
      variable = "aws:ResourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
  statement {
    sid    = "EventBridgeManage"
    effect = "Allow"
    actions = [
      "events:DeleteRule",
      "events:DisableRule",
      "events:EnableRule",
      "events:PutRule",
      "events:PutTargets",
      "events:RemoveTargets",
    ]
    resources = ["arn:aws:events:*:*:rule/DO-NOT-DELETE-S3-Files*"]
    condition {
      test     = "StringEquals"
      variable = "events:ManagedBy"
      values   = ["elasticfilesystem.amazonaws.com"]
    }
  }
  statement {
    sid    = "EventBridgeRead"
    effect = "Allow"
    actions = [
      "events:DescribeRule",
      "events:ListRuleNamesByTarget",
      "events:ListRules",
      "events:ListTargetsByRule",
    ]
    resources = ["arn:aws:events:*:*:rule/*"]
  }
}
resource "aws_iam_role_policy" "s3_files_bucket_access" {
  name   = "s3-files-bucket-access"
  role   = aws_iam_role.s3_files.name
  policy = data.aws_iam_policy_document.s3_files_bucket_access.json
}

Step 2. S3 bucket, S3 Files filesystem, and mount targets

Create an S3 bucket with versioning enabled — this is required for S3 Files. Add a lifecycle policy to limit the number of retained versions and control storage costs.
Create the S3 Files filesystem and attach the IAM role created above.
Create a security group allowing NFS traffic (port 2049) from EKS nodes.
Create a mount target in each private subnet and attach the security group.
If using a custom path, create an S3 object to represent the directory.
Create an access point scoped to the custom path.

s3-files-bucket.tf

resource "aws_s3_bucket" "s3filesstore" {
  bucket = "s3filesstore-${module.eks.cluster_name}"
  tags = {
    Name        = "s3filesstore-${module.eks.cluster_name}"
    Environment = "test"
    Terraform   = "true"
  }
}
resource "aws_s3_bucket_public_access_block" "s3filesstore" {
  bucket                  = aws_s3_bucket.s3filesstore.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
# Versioning must be enabled on the bucket for S3 Files to work
resource "aws_s3_bucket_versioning" "s3filesstore" {
  bucket = aws_s3_bucket.s3filesstore.id
  versioning_configuration {
    status = "Enabled"
  }
}
# Delete non-current versions older than 7 days to control storage costs
resource "aws_s3_bucket_lifecycle_configuration" "s3filesstore" {
  bucket = aws_s3_bucket.s3filesstore.id

  rule {
    id     = "expire-noncurrent-versions"
    status = "Enabled"

    noncurrent_version_expiration {
      noncurrent_days = 7
    }
  }
}
resource "aws_s3files_file_system" "s3filesstore" {
  bucket   = aws_s3_bucket.s3filesstore.arn
  role_arn = aws_iam_role.s3_files.arn
  tags = {
    Name        = "s3filesstore-${module.eks.cluster_name}"
    Environment = "test"
    Terraform   = "true"
  }
}
resource "aws_security_group" "s3files" {
  name        = "s3files-${module.eks.cluster_name}"
  description = "Allow NFS from EKS nodes to S3 Files"
  vpc_id      = module.vpc.vpc_id
  ingress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    security_groups = [module.eks.node_security_group_id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
resource "aws_s3files_mount_target" "s3filesstore" {
  count           = length(module.vpc.private_subnets)
  file_system_id  = aws_s3files_file_system.s3filesstore.id
  subnet_id       = module.vpc.private_subnets[count.index]
  security_groups = [aws_security_group.s3files.id]
}
# Skip the resources below if a custom mount path is not required
# Create an S3 object to represent the demoapp directory (required for custom access point)
resource "aws_s3_object" "demoapp" {
  bucket  = aws_s3_bucket.s3filesstore.id
  key     = "demoapp/"
  content = ""
}
# Create access point scoped to the demoapp directory
resource "aws_s3files_access_point" "demoapp" {
  file_system_id = aws_s3files_file_system.s3filesstore.id
  root_directory {
    path = "/demoapp"
  }
}

Mount S3 Files in a Pod

With the filesystem and access point in place, you are ready to mount S3 Files in a pod. The EFS folder in the repo contains two manifests:

File	Purpose
`s3-storageclass.yaml`	Creates a StorageClass named `s3files-sc` using the `efs-csi-driver`
`pod-s3.yaml`	Creates a PersistentVolume (using the CSI driver and volume handle), a PVC, and a pod that mounts the volume

kubectl apply -f s3-storageclass.yaml
kubectl apply -f pod-s3.yaml