Prometheus Architecture & PromQL Reference

Prometheus is an open-source systems monitoring and alerting toolkit that collects and stores metrics as time-series data identified by metric names and key/value label pairs.

---

Metric Types

Type	Behavior	Example
Counter	Cumulative metric that only increases (resets to 0 on restart).	http_requests_total, node_cpu_seconds_total
Gauge	Value that fluctuates up and down arbitrarily.	node_memory_Active_bytes, container_cpu_usage_seconds_total
Histogram	Samples observations (usually duration/size) and counts them in configurable buckets.	http_request_duration_seconds_bucket
Summary	Similar to Histogram, calculates configurable quantiles over a sliding time window.	rpc_duration_seconds{quantile="0.99"}

---

Scrape Configuration: Relabeling vs. Metric Relabeling

graph LR
    Target["Discovered Target"] --> Relabel["relabel_configs<br/>(Filters/Rewrites Target Labels before Scrape)"]
    Relabel --> Scrape["Scrape Endpoint"]
    Scrape --> MetricRelabel["metric_relabel_configs<br/>(Drops/Rewrites Time-Series Data before Storage)"]
    MetricRelabel --> TSDB["Prometheus TSDB"]

1. relabel_configs (Target Level)

Rewrites or filters targets before scraping occurs:

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    target_label: app
    action: replace

2. metric_relabel_configs (Metric Level)

Filters or drops specific metrics after scraping but before writing to TSDB to reduce high cardinality:

metric_relabel_configs:
  # Drop noisy Go runtime metrics
  - source_labels: [__name__]
    regex: "go_.*"
    action: drop

---

Essential PromQL Queries

# 1. Per-second rate of HTTP requests over a 5-minute window
rate(http_requests_total[5m])

# 2. Aggregated CPU utilization by namespace
sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)

# 3. 99th percentile request latency
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# 4. Top 10 metrics with the highest time-series cardinality
topk(10, count by (__name__)({__name__=~".+"}))