Prometheus Alertmanager Configuration & Routing

Alertmanager handles alerts sent by client applications such as Prometheus. It takes care of deduplicating, grouping, routing alerts to receivers (Opsgenie, PagerDuty, Slack, Email, MS Teams), and silencing/inhibiting cascaded alerts.

---

Core Alertmanager Concepts

graph TD
    Alerts["Prometheus Alerting Rules"] --> Grouping["Deduplication & Grouping<br/>(group_by: [alertname, cluster])"]
    Grouping --> Inhibition{"Inhibition Rules<br/>(Suppress Warning if Critical is Active)"}
    Inhibition --> Routing["Routing Tree<br/>(match_labels)"]
    Routing --> Receivers["Receivers<br/>(Opsgenie, Slack, Email, Webhooks)"]

• Grouping: Bundles multiple similar alerts into a single notification to avoid alert storms during major outages.
• Inhibition: Suppresses notifications for child alerts if a root-cause parent alert is already firing (e.g. mute node disk warnings if the entire node is offline).
• Silencing: Mutes specific alerts during planned maintenance windows based on label matchers.

---

Complete alertmanager.yaml Configuration

global:
  resolve_timeout: 5m

# 1. Routing Tree
route:
  receiver: "slack-notifications"
  group_by: ["alertname", "cluster", "namespace"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical alerts route to PagerDuty / Opsgenie
    - matchers:
        - severity = critical
      receiver: "pagerduty-critical"
      continue: true

    # Webapp alerts route to dev team webhook
    - matchers:
        - team = webapp
      receiver: "webteam-webhook"

# 2. Inhibition Rules
inhibit_rules:
  # Inhibit warning alerts if a critical alert for the same service is firing
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: ["alertname", "service", "namespace"]

# 3. Notification Receivers
receivers:
  - name: "slack-notifications"
    slack_configs:
      - channel: "#alerts"
        api_url: "https://hooks.slack.com/services/T00/B00/X00"
        send_resolved: true

  - name: "pagerduty-critical"
    pagerduty_configs:
      - routing_key: "pd-service-key"
        send_resolved: true

  - name: "webteam-webhook"
    webhook_configs:
      - url: "https://api.vettom.pages.dev/webhook/alerts"
        send_resolved: true