Chapter 38: Observability

Observability is the ability to understand the internal state of a system from its external outputs. Unlike monitoring (which answers "is the system working?"), observability answers "why is the system behaving this way?" through three pillars: metrics, logs, and traces.

1. Three Pillars of Observability

The three pillars provide complementary views into system behavior:

Metrics: Numerical measurements over time (CPU usage, request rate, error count)
Logs: Discrete events with context (request received, error occurred, DB query executed)
Traces: Request flow through distributed systems (showing latency and dependencies)

graph TB Alert[Alert Triggered: High Latency] Metrics[Metrics: Identify When] Traces[Traces: Identify Where] Logs[Logs: Identify Why] Alert --> Metrics Metrics --> |P99 latency spike
on /api/orders| Traces Traces --> |Slow span:
payment-service 2.5s| Logs Logs --> |ERROR: DB connection
pool exhausted| Root Root[Root Cause Found] style Alert fill:#ff6b6b,color:#1a1a2e style Metrics fill:#4ecdc4,color:#1a1a2e style Traces fill:#95e1d3,color:#1a1a2e style Logs fill:#f9ca24 style Root fill:#6c5ce7

Interview Tip: When asked about debugging production issues, walk through the three pillars: "I'd start with metrics to narrow down when and what, use traces to find where in the distributed system, then dive into logs for the specific error context."

How the Pillars Relate

Pillar	Cardinality	Storage Cost	Query Pattern	Use Case
Metrics	Low	Very cheap	Aggregation over time	Dashboards, alerting, trends
Logs	High	Expensive	Full-text search, filtering	Debugging, audit trails
Traces	Medium	Moderate (sampled)	Request-scoped queries	Performance analysis, dependency mapping

Correlation: Modern observability links the pillars via identifiers (trace_id, span_id) embedded in logs and metrics, enabling seamless navigation from alert to root cause.

2. Metrics

Metric Types

Type	Description	Example	Operations
Counter	Monotonically increasing value (resets on restart)	http_requests_total, errors_total	rate(), increase()
Gauge	Point-in-time value that can go up or down	cpu_usage, memory_bytes, queue_depth	avg(), max(), min()
Histogram	Distribution of values in buckets (client-side)	request_duration_seconds (buckets: 0.1, 0.5, 1, 5)	histogram_quantile()
Summary	Quantiles calculated on client (expensive)	request_duration_seconds{quantile="0.99"}	Direct percentile access

Histogram vs Summary: Histograms allow server-side aggregation across instances (use for latency). Summaries calculate quantiles on the client (cannot aggregate, but exact percentiles).

Golden Signals (Google SRE)

Latency: Time to serve a request (distinguish success vs error latency)
Traffic: Demand on the system (requests/sec, bandwidth)
Errors: Rate of failed requests (explicit failures + implicit failures like wrong content)
Saturation: How "full" the system is (CPU, memory, I/O, queue depth)

USE Method (Resources)

For every resource (CPU, disk, network), measure:

Utilization: % time resource is busy
Saturation: Queue depth or wait time
Errors: Count of error events

RED Method (Services)

For every service, measure:

Rate: Requests per second
Errors: Error rate (failures per second)
Duration: Latency distribution (P50, P95, P99)

Interview Question: "What metrics would you track for a payment API?"
Answer: "RED method: request rate (total and per endpoint), error rate (4xx vs 5xx), latency (P50/P95/P99). Golden signals: add saturation (DB connection pool, queue depth). Business metrics: successful payments, revenue processed. Dependency metrics: external payment gateway latency and error rate."

3. Prometheus

Prometheus is a pull-based time-series database and monitoring system, originally built at SoundCloud.

Pull Model

Unlike push-based systems (StatsD, CloudWatch), Prometheus scrapes metrics from HTTP endpoints (/metrics) at regular intervals (default 15s). Benefits:

Centralized control of scrape targets (service discovery)
Detect down targets (missed scrapes)
No agent configuration on each service

Architecture

PromQL

Powerful query language for selecting and aggregating time-series data:

# Request rate per second (5-minute average)
rate(http_requests_total[5m])

# P95 latency by status code
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
  by (le, status)
)

# Error rate as percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# CPU usage by pod (top 5)
topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))
            

Recording Rules

Pre-compute expensive queries for dashboard performance:

groups:
  - name: api_metrics
    interval: 30s
    rules:
      - record: api:request_rate:5m
        expr: sum(rate(http_requests_total[5m])) by (service, endpoint)

      - record: api:error_rate:5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            

Alerting Rules

groups:
  - name: latency_alerts
    rules:
      - alert: HighLatency
        expr: histogram_quantile(0.95, http_request_duration_seconds_bucket) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency on {{ $labels.service }}"
          description: "P95 latency is {{ $value }}s (threshold: 2s)"
            

Federation

Hierarchical Prometheus deployments: regional Prometheus servers scrape local targets, global Prometheus scrapes aggregated metrics from regional servers using the /federate endpoint.

Interview Tip: Mention Prometheus limitations: no long-term storage (use Thanos or Cortex), no built-in authentication (use reverse proxy), pull model requires network access to targets (Pushgateway for batch jobs).

4. Grafana

Grafana is a multi-datasource visualization platform for building dashboards and alerting.

Dashboards

Key panel types:

Time series: Line/area charts for metrics over time
Stat: Single value with threshold colors
Gauge: Progress toward limit (CPU usage, queue depth)
Heatmap: Latency distribution over time
Table: Structured data with sorting/filtering
Logs: Log stream from Loki or Elasticsearch

Variables & Templating

Create dynamic dashboards with dropdown selectors:

# Variable: environment (query)
label_values(up, environment)

# Variable: service (query, depends on $environment)
label_values(up{environment="$environment"}, service)

# Use in query:
rate(http_requests_total{service="$service", env="$environment"}[5m])
            

Alerting

Grafana 8+ has unified alerting (replacing legacy dashboard alerts):

Multi-datasource alerts (Prometheus + Loki)
Alert rules grouped by folders
Notification policies with routing (by label matchers)
Silences and maintenance windows

Data Sources

Grafana supports 100+ data sources: Prometheus, Loki, Elasticsearch, InfluxDB, CloudWatch, BigQuery, Jaeger, Tempo, PostgreSQL, MySQL, and more. Single dashboard can query multiple sources.

SLO Dashboards

Service Level Objective dashboards track error budget:

# SLO: 99.9% availability (43m downtime/month)
# Error budget: 0.1% = 43 minutes

# Current availability (30-day window)
1 - (
  sum(rate(http_requests_total{status=~"5.."}[30d]))
  /
  sum(rate(http_requests_total[30d]))
)

# Error budget remaining (%)
(target_availability - actual_availability) / (1 - target_availability) * 100
            

Dashboard Best Practices: Use RED/USE methods, align metrics with SLOs, add annotations for deploys/incidents, limit to 1-2 time ranges, use consistent color schemes, add links to runbooks.

5. Logging

Structured Logging

Log as JSON for machine readability and efficient querying:

// Bad: unstructured string
logger.info("User login failed for user@example.com from 192.168.1.1")

// Good: structured JSON
logger.info("User login failed", {
  "event": "auth.login.failed",
  "user_email": "user@example.com",
  "ip": "192.168.1.1",
  "trace_id": "a3f8b2c1",
  "timestamp": "2026-05-12T14:32:11Z",
  "duration_ms": 45
})
            

Log Levels

Level	Usage	Production Volume
ERROR	Application error requiring attention	Low (should alert)
WARN	Unexpected but handled (retry, fallback)	Low-Medium
INFO	Significant events (startup, request handling)	Medium (default prod level)
DEBUG	Detailed diagnostic info	High (dev/staging only)
TRACE	Very verbose (loop iterations, variable values)	Very High (disabled in prod)

Correlation IDs

Trace requests across services by propagating IDs:

# Request enters system
X-Request-ID: 7f8a3b2c
X-Trace-ID: a1b2c3d4  # From OpenTelemetry
X-Span-ID: e5f6a7b8

# Every log entry includes:
{
  "trace_id": "a1b2c3d4",
  "span_id": "e5f6a7b8",
  "message": "Processing payment"
}

# Query: show all logs for trace_id=a1b2c3d4
# Result: complete request flow across all services
            

ELK Stack

Elasticsearch: Distributed search engine, stores logs as JSON documents with full-text indexing
Logstash: Data pipeline for parsing, filtering, enriching logs (grok patterns for unstructured logs)
Kibana: UI for searching logs, building dashboards, creating alerts
Beats: Lightweight shippers (Filebeat for logs, Metricbeat for metrics)

Loki (Grafana)

Like "Prometheus for logs" - only indexes labels, not full-text content. Much cheaper storage than Elasticsearch, designed for logs already structured with labels (Kubernetes).

# LogQL query (similar to PromQL)
{namespace="production", service="payment-api"}
  |= "error"
  | json
  | status_code >= 500
  | line_format "{{.timestamp}} {{.message}}"
            

Interview Tip: Compare ELK vs Loki: "ELK is better for full-text search and complex queries. Loki is cheaper and faster for label-based queries, ideal for cloud-native apps with structured logs."

Log Retention & Sampling

Hot tier: Last 7 days, full logs, fast SSD storage
Warm tier: 8-30 days, slower HDD, compressed
Cold tier: 30+ days, object storage (S3), rarely queried
Sampling: Keep 100% of errors, sample INFO logs (e.g., 10% of successful requests)

6. Distributed Tracing

Distributed tracing tracks requests as they flow through microservices, showing latency and dependencies.

Concepts

Trace: End-to-end journey of a request (unique trace_id)
Span: Single operation within a trace (API call, DB query, cache lookup) with start time and duration
Parent-Child Relationship: Spans form a tree (parent span calls multiple child spans)
Span Context: Metadata propagated between services (trace_id, span_id, sampling decision)

graph TB subgraph "Trace: 7a3f9b2c (Total: 450ms)" Root[API Gateway
POST /orders
450ms] Auth[Auth Service
validate token
50ms] Order[Order Service
create order
380ms] DB1[PostgreSQL
insert order
120ms] Inv[Inventory Service
check stock
200ms] DB2[Redis
get inventory
15ms] Pay[Payment Service
charge card
180ms] Ext[Stripe API
charge
150ms] end Root --> Auth Root --> Order Order --> DB1 Order --> Inv Inv --> DB2 Order --> Pay Pay --> Ext style Root fill:#e85d75 style Order fill:#4ecdc4,color:#1a1a2e style Inv fill:#95e1d3,color:#1a1a2e style Pay fill:#f9ca24

Context Propagation

W3C Trace Context (Standard)

# HTTP headers
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
              ││ └─ trace id (128-bit)              └─ span id  └─ flags
              │└─ version

tracestate: vendor1=value1,vendor2=value2
            

B3 Propagation (Zipkin)

# HTTP headers (multi-header format)
X-B3-TraceId: 0af7651916cd43dd8448eb211c80319c
X-B3-SpanId: b7ad6b7169203331
X-B3-ParentSpanId: 00f067aa0ba902b7
X-B3-Sampled: 1
            

How Traces are Assembled

sequenceDiagram participant Client participant API as API Gateway participant Order as Order Service participant Collector as Trace Collector participant Storage as Trace Storage Client->>API: POST /orders (trace_id: 123) API->>API: Create root span API->>Order: Call order service (trace_id: 123, parent_span: A) Order->>Order: Create child span B Order->>Collector: Send span B (async) Order-->>API: Response API->>Collector: Send span A (async) API-->>Client: Response Collector->>Storage: Write spans Storage->>Storage: Assemble trace 123
(all spans with matching trace_id)

Services send spans asynchronously to a collector. The backend assembles the full trace by grouping spans with the same trace_id and building the tree using parent-child relationships.

Sampling: Tracing every request is expensive. Sampling strategies:

Head-based: Decision at trace start (sample 1% of requests)
Tail-based: Decision after trace completes (keep all errors, sample successful requests)

7. Jaeger & Zipkin

Jaeger (Uber) and Zipkin (Twitter) are distributed tracing backends. Jaeger has become the de facto standard, especially in cloud-native environments.

Jaeger Architecture

graph LR App[Applications
instrumented with
OpenTelemetry SDK] Agent[Jaeger Agent
receive spans
UDP/HTTP] Collector[Jaeger Collector
process + validate
write to storage] Storage[(Storage
Cassandra/Elasticsearch
Kafka buffer)] Query[Jaeger Query
API server] UI[Jaeger UI
trace visualization] App -->|send spans| Agent Agent -->|forward| Collector Collector -->|write| Storage Query -->|read| Storage UI -->|HTTP API| Query style Collector fill:#00d9ff style Storage fill:#60a5fa style UI fill:#f9ca24

Jaeger Agent: Sidecar process on each host, receives spans via UDP (low overhead) or HTTP, batches and forwards to collector
Jaeger Collector: Validates, indexes, and stores spans. Stateless, can scale horizontally
Storage: Cassandra (scalable), Elasticsearch (full-text search), or Kafka (buffer before storage)
Jaeger Query: API service for retrieving traces
Jaeger UI: Web interface for searching and visualizing traces

Zipkin

Similar architecture but simpler: no agent (apps send directly to collector). Storage options: Cassandra, Elasticsearch, MySQL. Jaeger supports Zipkin protocol for compatibility.

Sampling Strategies

Strategy	How It Works	Pros	Cons
Probabilistic (Head)	Sample X% at trace start	Simple, low overhead	May miss rare errors
Rate Limiting (Head)	Sample N traces/sec per service	Predictable volume	May miss bursts
Tail-based	Decide after trace completes	Keep all errors, slow requests	Requires buffering (complex, expensive)
Adaptive	Adjust rate based on error rate	Captures anomalies	Complex to tune

Interview Question: "How would you debug a latency spike using Jaeger?"
Answer: "Search for traces in the time window with high duration. Identify the slowest spans (critical path). Check if latency is in one service or propagated from a dependency. Look for sequential calls that could be parallelized. Check span tags for error indicators. Correlate with logs using trace_id."

Trace Visualization

Jaeger UI shows traces as Gantt charts (timeline view) and dependency graphs (service topology). Key features:

Span timeline with color coding for services
Critical path highlighting (slowest sequential chain)
Span tags and logs (structured metadata)
Trace comparison (compare slow vs fast traces)
Deep links to logs (via trace_id)

8. OpenTelemetry

OpenTelemetry (OTel) is a CNCF project providing a unified standard for observability: metrics, logs, and traces with a single SDK and instrumentation framework.

Why OpenTelemetry?

Before OTel, each backend required its own SDK (Jaeger SDK, Prometheus client, custom logging). OTel provides:

Single SDK for all three pillars (metrics, logs, traces)
Vendor-neutral standard (switch backends without code changes)
Auto-instrumentation for popular frameworks (Express, Django, Spring Boot)
Consistent context propagation (trace_id in logs and metrics)

Components

graph TB App[Application
OTel SDK] AutoInst[Auto-Instrumentation
frameworks, libraries] ManualInst[Manual Instrumentation
custom spans, metrics] Collector[OTel Collector
receive, process, export] Prom[Prometheus] Jaeger[Jaeger] Loki[Loki] Cloud[Cloud Backends
Datadog, New Relic, Honeycomb] App --> AutoInst App --> ManualInst App -->|OTLP| Collector Collector -->|Prometheus remote write| Prom Collector -->|Jaeger protocol| Jaeger Collector -->|Loki push| Loki Collector -->|OTLP| Cloud style App fill:#425cc7 style Collector fill:#f5a623 style Prom fill:#e85d75 style Jaeger fill:#00d9ff

Auto-Instrumentation

Zero-code instrumentation for common frameworks:

# Node.js (auto-instrument Express, HTTP, Redis, PostgreSQL)
require('@opentelemetry/auto-instrumentations-node/register');

# Python (auto-instrument Django, Flask, requests, psycopg2)
opentelemetry-instrument python app.py

# Java (auto-instrument Spring Boot, JDBC, HTTP clients)
java -javaagent:opentelemetry-javaagent.jar -jar app.jar
            

OTLP Protocol

OpenTelemetry Protocol (OTLP) is the standard wire format (gRPC or HTTP) for sending telemetry data from SDKs to collectors or backends.

# Export to OTel Collector
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=payment-api
OTEL_RESOURCE_ATTRIBUTES=environment=production,region=us-east-1
            

Collector Pipeline

The OTel Collector receives, processes, and exports telemetry. Pipeline stages:

Receivers: Ingest data (OTLP, Jaeger, Prometheus, logs)
Processors: Transform data (batch, filter, sample, add attributes, tail-sample)
Exporters: Send data to backends (Prometheus, Jaeger, Loki, cloud vendors)

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

  attributes:
    actions:
      - key: environment
        value: production
        action: insert

  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      - name: sample-success
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

exporters:
  prometheus:
    endpoint: 0.0.0.0:9090
  jaeger:
    endpoint: jaeger:14250
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, tail_sampling, attributes]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [loki]
            

Interview Tip: Emphasize OTel's vendor neutrality: "With OpenTelemetry, we can switch from self-hosted Jaeger to a managed service like Honeycomb without changing application code - just reconfigure the collector's exporters."

Semantic Conventions

OTel defines standard attribute names for common scenarios (HTTP requests, DB queries, messaging) to ensure consistency across services and vendors:

# HTTP span attributes
http.method = "POST"
http.url = "https://api.example.com/orders"
http.status_code = 200
http.response_content_length = 1024

# Database span attributes
db.system = "postgresql"
db.name = "orders_db"
db.statement = "SELECT * FROM orders WHERE user_id = ?"
db.connection_string = "postgresql://db:5432/orders_db"
            

9. Alerting

Alert Fatigue

Too many alerts lead to ignored pages and missed incidents. Causes:

Alerting on symptoms, not impact (disk 80% full vs service down)
No grouping or deduplication (100 hosts down = 100 alerts)
Noisy alerts (flapping, false positives)
Alerts without clear action (what should on-call do?)

Solution: Alert on SLO violations (user-impacting issues), add context (runbook links), group/dedupe, tune thresholds.

Alerting Best Practices

Principle	Description	Example
Alert on SLOs	Page when error budget is at risk	99.9% SLO with 5% budget burn rate (will breach in 20h)
Severity Levels	Critical (page), Warning (ticket), Info (log)	Critical: service down. Warning: latency +20%
Runbooks	Every alert links to resolution steps	"Check DB connection pool, restart if exhausted"
Grouping	Aggregate related alerts (by cluster, service)	10 pods down → "Cluster outage" (not 10 pages)
Deduplication	Suppress repeat alerts within time window	Page once, send updates every 15min

Routing

graph TB Alert[Alert Fired] AM[Alertmanager] Sev{Severity?} Team{Team?} Page[PagerDuty
On-call Engineer] Slack[Slack #alerts] Ticket[Jira Ticket] Alert --> AM AM --> Sev Sev -->|critical| Page Sev -->|warning| Team Sev -->|info| Slack Team -->|platform| Page Team -->|backend| Ticket style Alert fill:#ff6b6b,color:#1a1a2e style Page fill:#e85d75 style Slack fill:#4ecdc4,color:#1a1a2e style Ticket fill:#f9ca24

Alertmanager Routing

route:
  receiver: default
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: pagerduty
      continue: true

    - match:
        severity: warning
        team: platform
      receiver: platform-slack

    - match:
        severity: warning
      receiver: jira

receivers:
  - name: pagerduty
    pagerduty_configs:
      - service_key: "abc123"

  - name: platform-slack
    slack_configs:
      - channel: "#platform-alerts"
        text: "{{ .CommonAnnotations.summary }}"

  - name: jira
    webhook_configs:
      - url: "https://jira.example.com/webhook"
            

SLO-Based Alerting (Burn Rate)

Alert when error budget is consumed too quickly. For a 99.9% SLO (43min downtime/month):

# Fast burn (page immediately): 5% budget consumed in 1 hour
- alert: ErrorBudgetFastBurn
  expr: |
    (
      1 - (sum(rate(http_requests_total{status!~"5.."}[1h]))
           / sum(rate(http_requests_total[1h])))
    ) > (1 - 0.999) * 5
  for: 2m
  annotations:
    summary: "Error budget burning 5x faster than expected"

# Slow burn (warning): 10% budget consumed in 24 hours
- alert: ErrorBudgetSlowBurn
  expr: |
    (
      1 - (sum(rate(http_requests_total{status!~"5.."}[24h]))
           / sum(rate(http_requests_total[24h])))
    ) > (1 - 0.999) * 0.1
  for: 15m
  labels:
    severity: warning
            

Interview Question: "How do you balance alert coverage vs alert fatigue?"
Answer: "Focus on user-facing SLO violations for pages. Use multi-window, multi-burn-rate alerts to detect both fast and slow issues. Add lower-severity alerts for early warning signals (e.g., latency trending up). Regularly review alert effectiveness: track MTTD (mean time to detect), false positive rate, and alert actionability. If alerts fire but no action is taken, they should be removed or demoted."

Runbooks

Every alert should include:

Impact: What is broken? Who is affected?
Diagnosis: How to confirm the alert is accurate?
Mitigation: Immediate steps to restore service
Resolution: Long-term fix
Links: Dashboards, logs, traces, escalation contacts

10. Observability in Practice

Dashboard Design

Effective dashboards follow a hierarchy:

Top-level: System health at a glance (RED/Golden Signals, SLO compliance)
Service-level: Per-service metrics (request rate, latency, errors, dependencies)
Resource-level: Host metrics (CPU, memory, disk, network)
Deep-dive: Detailed views for investigation (per-endpoint latency, error types)

Layout Tips:

Most important metrics at the top (SLO compliance, error rate)
Use consistent time ranges (align all panels)
Add annotations for deploys, incidents, config changes
Limit to 1-2 dashboards per service (avoid dashboard sprawl)
Link to runbooks, logs, traces from panels

On-Call Workflow

graph TB Alert[Alert Received
PagerDuty/Slack] Ack[Acknowledge Alert
Start timer] Dash[Check Dashboard
Validate alert] Valid{Valid Alert?} FalsePos[Mark False Positive
Silence + ticket to fix] Impact{User Impact?} Incident[Declare Incident
Page additional responders] Investigate[Investigate
Metrics → Traces → Logs] RootCause{Root Cause?} Mitigate[Mitigate
Rollback/restart/scale] Resolved[Verify Resolved
Update SLO dashboard] Postmortem[Postmortem
Document + action items] Alert --> Ack Ack --> Dash Dash --> Valid Valid -->|No| FalsePos Valid -->|Yes| Impact Impact -->|High| Incident Impact -->|Low| Investigate Incident --> Investigate Investigate --> RootCause RootCause -->|Yes| Mitigate RootCause -->|No| Investigate Mitigate --> Resolved Resolved --> Postmortem style Alert fill:#ff6b6b,color:#1a1a2e style Incident fill:#e85d75 style Mitigate fill:#f9ca24 style Resolved fill:#6c5ce7

Incident Investigation Flow

graph LR Start[Alert: High Latency] M1[Metrics
When did it start?
Which endpoints?] M2[Metrics
Which services?
Error rate spike?] T1[Traces
Find slow requests] T2[Traces
Identify slow span] L1[Logs
Search by trace_id] L2[Logs
Find error message] RC[Root Cause:
DB connection pool exhausted] Start --> M1 M1 --> M2 M2 --> T1 T1 --> T2 T2 --> L1 L1 --> L2 L2 --> RC style Start fill:#ff6b6b,color:#1a1a2e style M1 fill:#4ecdc4,color:#1a1a2e style M2 fill:#4ecdc4,color:#1a1a2e style T1 fill:#95e1d3,color:#1a1a2e style T2 fill:#95e1d3,color:#1a1a2e style L1 fill:#f9ca24 style L2 fill:#f9ca24 style RC fill:#6c5ce7

Observability Decision Tree

graph TB Q1{What do I need?} Q1 -->|Track system health| Metrics Q1 -->|Debug specific request| Traces Q1 -->|Understand error details| Logs Metrics --> Q2{What kind?} Q2 -->|Request success/latency| RED[RED Method
rate, errors, duration] Q2 -->|Resource utilization| USE[USE Method
utilization, saturation, errors] Q2 -->|Business KPIs| Custom[Custom Metrics
revenue, signups, conversions] Traces --> Q3{Performance or dependencies?} Q3 -->|Slow request| Jaeger[Jaeger/Tempo
Identify slow span] Q3 -->|Service map| Topo[Topology View
Understand call graph] Logs --> Q4{Structured?} Q4 -->|Yes| Loki[Loki
Fast label queries] Q4 -->|No| ELK[ELK Stack
Full-text search] Q4 -->|Both| Hybrid[Loki + Elasticsearch
Cheap + searchable] style Q1 fill:#e85d75 style Metrics fill:#4ecdc4,color:#1a1a2e style Traces fill:#95e1d3,color:#1a1a2e style Logs fill:#f9ca24

Key Metrics to Track

Layer	Key Metrics	Tools
Business	Revenue, active users, conversion rate, feature adoption	Custom metrics, BigQuery
Application	Request rate, latency (P50/P95/P99), error rate, SLO compliance	Prometheus, Grafana
Dependencies	DB query time, cache hit rate, API latency (Stripe, Auth0)	OpenTelemetry, Jaeger
Infrastructure	CPU, memory, disk I/O, network bandwidth, pod restarts	Prometheus node_exporter, kube-state-metrics

Interview Question: "Design an observability stack for a microservices platform."
Answer: "I'd use OpenTelemetry for instrumentation (single SDK for metrics, logs, traces). Deploy OTel Collectors as a central pipeline for processing and routing. For storage: Prometheus + Thanos for metrics (long-term retention), Loki for logs (cost-effective, integrated with Grafana), Jaeger or Tempo for traces. Grafana for dashboards and alerting. Alertmanager for routing to PagerDuty and Slack. Key decisions: tail-based sampling in the collector to keep all errors while sampling successful requests, structured logging with trace_id for correlation, SLO-based alerts to avoid fatigue."

Observability Maturity Levels

Level 1: Basic monitoring (CPU, memory, uptime checks)
Level 2: Structured logging, centralized log aggregation (ELK)
Level 3: Metrics with dashboards (Prometheus + Grafana), basic alerting
Level 4: Distributed tracing (Jaeger), correlated logs/metrics/traces
Level 5: OpenTelemetry adoption, SLO-based alerting, automated incident response

Common Pitfalls

Cardinality explosion: High-cardinality labels (user_id, trace_id) in metrics cause memory issues. Solution: use logs/traces for high-cardinality data.
Missing correlation IDs: Cannot trace requests across services. Solution: propagate trace_id in all logs and metrics.
Over-sampling traces: Expensive storage, miss rare errors. Solution: tail-based sampling (keep errors, sample successes).
No runbooks: Alerts fire but responders don't know what to do. Solution: link every alert to a runbook.
Dashboard sprawl: 100 dashboards, nobody uses them. Solution: maintain 1-2 dashboards per service, deprecate old ones.

← Chapter 37 Table of Contents Chapter 39 →