Observability with OpenTelemetry (OTel)

OpenTelemetry (OTel) is a Cloud Native Computing Foundation (CNCF) incubating project that standardizes the generation, collection, processing, and exporting of vendor-neutral telemetry data (Metrics, Logs, and Traces).

---

The Three Pillars of Observability

graph TD
    App["Application Workload"] --> Traces["Traces (End-to-end distributed request flows)"]
    App --> Metrics["Metrics (Numeric aggregations over time: RPS, CPU, Latency)"]
    App --> Logs["Logs (Timestamped events and error details)"]

    Traces --> Collector["OpenTelemetry Collector"]
    Metrics --> Collector
    Logs --> Collector

    Collector --> Backends["Observability Backends<br/>(Prometheus, Jaeger, Grafana Tempo, Loki, CloudWatch)"]

---

OpenTelemetry Collector Pipeline

The OpenTelemetry Collector consists of four core pipeline stages:

Receivers ➔ Processors ➔ Exporters ➔ Extensions

• Receivers (Push/Pull): Ingest telemetry in OTLP (gRPC/HTTP), Jaeger, Zipkin, or Prometheus formats.
• Processors (Transform): Batch requests, add resource attributes (k8s pod name, cluster ID), filter out sensitive data, and sample traces.
• Exporters (Send): Forward formatted telemetry to backends (e.g. OTLP, Prometheus remote-write, Elasticsearch).
• Extensions (Health/Auth): Health checks, pprof diagnostics, and z-pages.

---

Collector Configuration Example (otel-collector-config.yaml)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  otlp:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

---

Auto-Instrumentation vs. Manual Instrumentation

Feature	Auto-Instrumentation (OTel Operator)	Manual Instrumentation
Code Changes	0 lines of application code	Explicit SDK imports & span annotations
Mechanism	Bytecode injection / Monkey patching at runtime	Explicit tracer.Start(ctx, "OperationName")
Best Used For	Standard HTTP/gRPC frameworks, database drivers	Fine-grained business logic & custom workflows