Ingress and HTTPRoutes as Prometheus HTTP_SD

Task: : Automate monitoring of Ingress and HTTPRoutes using Prometheus HTTP_SD

Solution: : Install httproute-prometheus-shim to expose Ingress and HTTPRoutes as Prometheus HTTP_SD] targets. This can the be consumed by monitoring tools like Prometheus.

App exposes via 2 URLs:

• /httproutes-sd : HTTPRoutes as Prometheus HTTP_SD targets
• /ingress-sd : Ingress as Prometheus HTTP_SD targets

Installing httproute-prometheus-shim

Application image and Helm chart are available in Docker registry dennysv/httproute-prometheus-shim/general

Prepare values file

# Optional HTTP route to expose it externally
httpRoute:
  enabled: false
  gatewayName: "default"
  gatewayNamespace: "gateway-system"
  hostnames: [promsd.vettom.online]
  sectionName: ""

Install with Helm chart

helm install httproute-prometheus-shim oci://registry-1.docker.io/dennysv/httproute-prometheus-shim \
 --version 1.0.3 -f values.yaml -n monitoring --create-namespace

Verify installation

If HTTP Route not enabled, port forward and check

kubectl port-forward <pod-name> 9113:9113
curl localhost:9113

Sample output:

[
  {
    "labels": {
      "__meta_application": "httproutes",
      "__meta_origin": "k8s_cluster",
      "__meta_source": "prom_shim_http_routes"
    },
    "targets": ["prometheus-blackbox.vettom.online", "argocd.vettom.online"]
  }
]

Automating monitoring

Configure prometheus to use the targets exposed by the httproute-prometheus-shim as targets for blackbox monitoring.

example for kube-prometheus-stack

additionalScrapeConfigs:
  - job_name: blackbox_httproutes-sd
    http_sd_configs:
      - url: http://httproute-prometheus-shim.monitoring.svc.cluster.local:9113/httproutes-sd
    metrics_path: /probe
    params:
      module:
        - http_2xx
    relabel_configs:
      - source_labels:
          - __address__
        target_label: __param_target
      - source_labels:
          - __param_target
        target_label: instance
      - source_labels: [__param_target]
        regex: "(.+)"
        replacement: "https://$1"
        target_label: __param_target
      - replacement: prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
        target_label: __address__