External secrets operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

In this example I will demonstrate configuring EKS with AWS Secrets manager and EKS Pod Identity. Current version of external-secrets (0.12.1) does have some limitations when it comes to SecretStore permission, so I will also demonstrate how to make use of IRSA for additional Secretstore. With Pod Identity, IAM policy is appliled to External secrets Pod and all SecretStores will use same policy, lacking granularity unless separate installations are created.

Install and configure External-secrets

1. Authentication: EKS Pod identity to access secrets
2. Install External-secrets Operator Helm chart
3. Configure SecretStore with access to secrets
4. Create ExternalSecret object to inject Kubernetes Secrets
5. Configure IAM RSA with access to specific secret
6. Create SecretStore with IAM RSA Authentication

Authentication : IAM policy and Pod Identity

Step 1. Create IAM policy

Create IAM policy that allow read access to SecretsManager secrets objects. In this example, I am allowing access to secrets under path eks/ in Secrets manager.

IAM policy esoControllerIAMPolicy

    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Effect" : "Allow",
          "Action" : [
            "secretsmanager:GetResourcePolicy",
            "secretsmanager:GetSecretValue",
            "secretsmanager:DescribeSecret",
            "secretsmanager:ListSecretVersionIds",
            "secretsmanager:GetRandomPassword",
            "secretsmanager:ListSecrets",
            "secretsmanager:BatchGetSecretValue"
          ],
          "Resource" : [
            "arn:aws:secretsmanager:eu-west-1:*:secret:eks/*"
          ]
        },                 
      ]
    }

Step 2. Create IAM role external-secrets with Pod Identity trust policy.

Trust policy for Pod Identity service

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:TagSession",
                "sts:AssumeRole"
            ]
        }
    ]
}

Step 3. Attache IAM policy esoControllerIAMPolicy to IAM role external-secrets

Step 4. Create Pod Identity association for Namespace externalsecrets, ServiceAccount external-secrets to the role created.

Step 5. Install External Secrets

Install External-Secrets to Namespace externalsecrets using Helm chart. We will use default values and default serviceAccount name is external-secrets which we have already associated with Pod Identity in previoud step.

 helm install external-secrets -n externalsecrets --create-namespace external-secrets \
   --repo https://charts.external-secrets.io --version  0.12.1

Step 6. Create SecretStore

You need to create a SecretStore or ClusterSecretStore to retrieve secrets from AWS Secrets Manager. I am creating ClusterSecretStore called eks-secretstore and restricting secrets access to specific Namespaces.

ClusterSecretStore eks-secretstore.yaml

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: eks-secretstore
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
  conditions:
      - namespaces:
        - default
        - cert-manager

When no Authentication details provided, Pod's identity will be used to fetch secrets. We have already configured Pod Identity for Externa-Secrets Service Account.

Step 7. Create ExternalSecret

Create ExternalSecret object and verify that secret can be accessed. PodIdentity IAM policy allows retrieving secrets from path starting eks/ ans ClusterSecretStore limits which Namespaces can be configured with secrets.

ClusterSecretStore eks-ext-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: mysecretfromsecretsmanager
  namespace: default
spec:
  refreshInterval: 4h
  secretStoreRef:
    name: eks-secretstore
    kind: ClusterSecretStore
  target:
    name: mysecretink8s
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: eks/mysecret
      property: user

• name: :Name of the ext-secret object in k8s
• secretStoreRef.name : Specify which Secretstore to use and its type
• target.name : Name of secret to be created on K8s
• data.secretKey : Store secret with in "username" key in secret mysecretink8s
• remoteRef.key : Path to secret in SecretsManager
• remoteRef.property : Represents secret value in SecretsManager

---

SecretStore's with IRSA

For some users Pod Identity with single policy may be enough. What if Developers need access to secrets in path starting "app/" This is where you can make use of IAM role for Service Account to create additional Secretstore's with own access policy.

Step 1. Create IAM role with IRSA auth policy

Below is a sample Trust policy for IRSA where it allows IAM user to authenticate with EKS OIDC provider. Assume role is restricted to namespace kube-system and service account apps-secrets

IRSA trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<aws account id>:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/8B2270E72361F3D8AA734727B"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-west-1.amazonaws.com/id/8B28D340FB61F3D8AA734727B:sub": "system:serviceaccount:kube-system:apps-secrets"
                }
            }
        }
    ]
}

Step 2. Create IAM policy and attach to role

Create IAM policy allowing access to secrets in path "app/*" and attach to IAM role.

Step 3. Create Service account in Kubernetes

Create a service account called apps-secrets in namespace `kube-system. Make sure to add annotation for role-arn

IRSA requires ServiceAccount to be annotated with ARN of the role to be assumed.

apps-serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<aws account id>:role/apps-secrets
  name: apps-secrets
  namespace: kube-system

Step 4. Create SecretStore for Apps

Create a secretstore and specify service account for Authentication

apps-secretstore.yaml

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: apps-secretstore
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: apps-secrets
            namespace: kube-system
  conditions:
      - namespaces:
        - default

Step 5. Verify by creating Secret

apps-ext-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myappssecret
  namespace: default
spec:
  refreshInterval: 4h
  secretStoreRef:
    name: apps-secretstore
    kind: ClusterSecretStore
  target:
    name: newsecret
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: apps/newsecret
      property: user

---