Prometheus High Availability & Long-Term Storage with Thanos

Prometheus stores metric data in a local TSDB on disk. For enterprise clusters with high metric volume and long retention requirements (e.g. 1+ years), running Prometheus standalone causes high RAM usage and disk bottlenecks.

Thanos transforms Prometheus into a horizontally scalable, highly available monitoring system with infinite object storage (Amazon S3), deduplication, and unified global querying across multiple clusters.

📁 Source Code: aws-eks-terraform / EKS-Prom-Thanos

---

📺 Video Walkthrough

---

Architecture Overview

graph TD
    subgraph EKS ["EKS Cluster"]
        Prom1["Prometheus Replica 0 + Thanos Sidecar"]
        Prom2["Prometheus Replica 1 + Thanos Sidecar"]
        Querier["Thanos Querier / Query-Frontend"]
        StoreGW["Thanos Store Gateway"]
        Compactor["Thanos Compactor"]
    end
    subgraph AWS ["Amazon S3"]
        S3Bucket["TSDB Historical Blocks (S3 Bucket)"]
    end

    Prom1 -- "Upload 2h TSDB Blocks" --> S3Bucket
    Prom2 -- "Upload 2h TSDB Blocks" --> S3Bucket
    Compactor -- "Deduplicate & Downsample" --> S3Bucket
    StoreGW -- "Read Historical Blocks" --> S3Bucket
    Querier -- "Realtime Metrics (0-2h)" --> Prom1
    Querier -- "Realtime Metrics (0-2h)" --> Prom2
    Querier -- "Historical Metrics (>2h)" --> StoreGW

---

Thanos Component Summary

Component	Role	Resource Profile
Thanos Sidecar	Runs alongside Prometheus; uploads TSDB blocks to S3 every 2 hours and answers realtime queries.	Lightweight sidecar container
Thanos Querier	Evaluates PromQL queries, deduplicates data across Prometheus replicas, and queries StoreGateways.	Stateless service
Thanos Query-Frontend	Caches query responses and splits heavy range queries into smaller day-by-day queries.	Stateless caching proxy
Thanos Store Gateway	Acts as a proxy to query historical TSDB blocks directly from Amazon S3.	Memory-intensive / Stateful
Thanos Compactor	Downsamples (5m and 1h resolutions) and compacts TSDB blocks in the S3 bucket.	Run as single replica

---

Step 1. S3 Bucket & IAM Policy with EKS Pod Identity

thanos-s3-iam.tf

resource "aws_s3_bucket" "thanos_store" {
  bucket = "eks-thanos-metrics-${module.eks.cluster_name}"
}

data "aws_iam_policy_document" "thanos_s3" {
  statement {
    effect = "Allow"
    actions = [
      "s3:ListBucket",
      "s3:PutObject",
      "s3:GetObject",
      "s3:DeleteObject"
    ]
    resources = [
      aws_s3_bucket.thanos_store.arn,
      "${aws_s3_bucket.thanos_store.arn}/*"
    ]
  }
}

resource "aws_iam_policy" "thanos_s3" {
  name   = "ThanosS3Policy-${module.eks.cluster_name}"
  policy = data.aws_iam_policy_document.thanos_s3.json
}

resource "aws_iam_role" "thanos" {
  name               = "thanos-role-${module.eks.cluster_name}"
  assume_role_policy = data.aws_iam_policy_document.pod_id_trust.json
}

resource "aws_iam_role_policy_attachment" "thanos" {
  role       = aws_iam_role.thanos.name
  policy_arn = aws_iam_policy.thanos_s3.arn
}

# Pod Identity associations for Sidecar, Store Gateway, and Compactor
resource "aws_eks_pod_identity_association" "thanos_sidecar" {
  cluster_name    = module.eks.cluster_name
  namespace       = "monitoring"
  service_account = "kube-prometheus-stack-prometheus"
  role_arn        = aws_iam_role.thanos.arn
}

resource "aws_eks_pod_identity_association" "thanos_store" {
  cluster_name    = module.eks.cluster_name
  namespace       = "thanos"
  service_account = "thanos-storegateway"
  role_arn        = aws_iam_role.thanos.arn
}

---

Step 2. Configure Prometheus with Thanos Sidecar

prometheus-thanos-values.yaml

prometheus:
  prometheusSpec:
    retention: 4h # Prometheus holds recent data; Thanos manages long-term S3 storage
    replicas: 2
    replicaExternalLabelName: "prometheus_replica"
    thanos:
      version: v0.39.1
      objectStorageConfig:
        secret:
          type: S3
          config:
            bucket: "eks-thanos-metrics-demo"
            region: "eu-west-1"
            endpoint: s3.amazonaws.com
            insecure: false

helm upgrade -i kube-prometheus-stack \
  --repo https://prometheus-community.github.io/helm-charts kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  -f prometheus-thanos-values.yaml

---

Step 3. Install Thanos Core Components via Helm

thanos-values.yaml

image:
  registry: docker.io
  repository: thanosio/thanos
  tag: v0.40.1

objstoreConfig:
  type: S3
  config:
    bucket: "eks-thanos-metrics-demo"
    region: "eu-west-1"
    endpoint: s3.amazonaws.com

query:
  enabled: true
  replicaLabel: prometheus_replica
  dnsDiscovery:
    enabled: true
    sidecarsService: kube-prometheus-stack-thanos-discovery
    sidecarsNamespace: monitoring

storegateway:
  enabled: true
  replicaCount: 2

compactor:
  enabled: true
  retentionResolutionRaw: 30d
  retentionResolution5m: 90d
  retentionResolution1h: 365d

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm install thanos bitnami/thanos \
  --namespace thanos \
  --create-namespace \
  -f thanos-values.yaml

---

Step 4. Grafana Datasource Configuration

Point your Grafana Prometheus data source URL to the Thanos Query frontend:

http://thanos-query-frontend.thanos.svc.cluster.local:9090

When Grafana runs dashboard queries: - Data from the last 0-2 hours is fetched live from Prometheus TSDB. - Historical data (>2 hours) is streamed dynamically from the S3 bucket through the Thanos Store Gateway.