Chapter 39: Distributed Tracing

1. Why Distributed Tracing

The Microservices Debugging Problem

In monolithic applications, debugging is straightforward: add logs, set breakpoints, follow the stack trace. In distributed systems, a single user request flows through dozens of services across multiple data centers. Traditional debugging breaks down:

• No single stack trace: Request spans 10+ services, each with its own logs
• Latency attribution: Total response time is 3 seconds. Which service is slow?
• Error correlation: Service D returns 500. Was it caused by Service B's timeout or Service C's invalid data?
• Cascading failures: One slow service causes timeouts in 5 others. Which is the root cause?

What Distributed Tracing Provides

Request Flows Across Services

Consider an e-commerce checkout request:

API Gateway → Frontend Service → Auth Service → Order Service → Inventory Service → Payment Service → Email Service

Problem: Response took 3.2s. Which service was slow?
Each service logs: "Request completed in 145ms" but doesn't track upstream latency.
No visibility into parallel calls, retries, or queue wait times.

Latency Attribution

Distributed tracing breaks down response time into components:

• Service time: Actual work done by each service
• Network time: Transmission between services
• Queue time: Waiting in message queues, thread pools
• Database time: Query execution, connection pool waits
• External API time: Third-party service calls

This attribution enables targeted optimization: if 80% of latency is database queries, focus there instead of optimizing application code.

2. Core Concepts

Trace

A trace represents the complete journey of a request through the system. It has a unique trace_id (128-bit or 64-bit) that is assigned at the entry point (API gateway, load balancer) and propagated to all downstream services.

Span

A span represents a single operation within a trace. Each span has:

• span_id: Unique identifier for this operation (64-bit)
• parent_span_id: Link to the calling operation (null for root span)
• operation name: Descriptive name (e.g., "db.query", "http.request")
• start time & duration: Timing data
• attributes/tags: Key-value metadata (HTTP status, error flag, user ID)
• events/logs: Timestamped annotations within the span

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "8e3c31d5baad7b8a",
  "operation_name": "payment.charge",
  "start_time": "2026-05-12T10:30:42.123Z",
  "duration_ms": 2045,
  "attributes": {
    "http.method": "POST",
    "http.status_code": 200,
    "payment.amount": 49.99,
    "payment.gateway": "stripe",
    "db.statement": "INSERT INTO payments..."
  },
  "events": [
    {"timestamp": "2026-05-12T10:30:43.500Z", "name": "gateway_request_sent"},
    {"timestamp": "2026-05-12T10:30:44.100Z", "name": "gateway_response_received"}
  ]
}

Span Context

Span context is the minimal data needed to link spans across process boundaries. It contains:

• trace_id: Identifies the trace
• span_id: Identifies the current span (becomes parent_span_id in child spans)
• trace_flags: Sampling decision, debug flag
• trace_state: Vendor-specific context (optional)

Span context is propagated via HTTP headers, message metadata, or RPC frameworks.

Parent-Child Relationships

Spans form a tree structure. The root span represents the entry point. Child spans represent nested operations:

The critical path (longest execution path) is A → C → E → G, totaling 2.8s. Optimizing other spans won't improve overall latency.

Baggage

Baggage is key-value data propagated across the entire trace. Unlike span attributes (stored per-span), baggage is inherited by all descendant spans. Use cases:

• User context: user_id, tenant_id, session_id
• Feature flags: Propagate experiment variants
• Request metadata: client_version, device_type
• Security context: auth scopes, permissions

Trace Waterfall Diagram

The canonical visualization for traces is a waterfall diagram, showing spans on a timeline:

Time (ms)    0        500      1000     1500     2000     2500     3000
             |--------|--------|--------|--------|--------|--------|
GET /checkout|================================================| 3245ms
  auth.verify|===|                                              150ms
  order.create    |==========================================| 2800ms
    inventory.check    |====|                                  400ms
    payment.charge     |================================|     2045ms
      db.query              |=|                                 35ms
      stripe.api                |=======================|     1980ms
    email.send                                    |==|          220ms
                    

3. Context Propagation

The Cross-Process Challenge

Traces span multiple processes, hosts, and languages. How does Service B know it's part of the same trace as Service A? Context propagation serializes span context and transmits it across boundaries via:

• HTTP headers: Most common for REST/GraphQL APIs
• Message metadata: Kafka headers, SQS attributes, RabbitMQ properties
• gRPC metadata: Key-value pairs in gRPC requests
• Function parameters: For in-process propagation

W3C TraceContext Standard

The W3C TraceContext specification defines two HTTP headers for context propagation:

traceparent Header

Encodes trace_id, span_id, and trace flags in a compact format:

tracestate Header

Optional vendor-specific context. Multiple vendors can append their data:

tracestate: aws=s:1;r:5;k:abc123,datadog=parent_id=123456789

This allows multiple tracing systems to coexist without interfering.

B3 Headers (Zipkin)

The older B3 propagation format uses separate headers:

X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId: 00f067aa0ba902b7
X-B3-ParentSpanId: 8e3c31d5baad7b8a
X-B3-Sampled: 1

Or single-header format:

b3: 4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-1-8e3c31d5baad7b8a

W3C TraceContext is preferred for new systems, but B3 is still widely used (Twitter, Uber, Netflix).

Injectors and Extractors

Tracing libraries provide injectors (write context to headers) and extractors (read context from headers):

// Injector: Service A sends request to Service B
const span = tracer.startSpan('http.request');
const headers = {};
tracer.inject(span.context(), FORMAT_HTTP_HEADERS, headers);

// headers now contains:
// { 'traceparent': '00-4bf92f...', ... }

fetch('https://service-b/api', { headers });

// Extractor: Service B receives request
const parentContext = tracer.extract(FORMAT_HTTP_HEADERS, request.headers);
const childSpan = tracer.startSpan('process.request', { childOf: parentContext });

Cross-Process Propagation

Context propagation happens at every service boundary:

Async and Queue Propagation

Context propagation is tricky for message queues and async tasks because there's no HTTP request/response:

• Kafka: Store traceparent in message headers
• SQS: Store traceparent in message attributes
• RabbitMQ: Store traceparent in message properties
• Background jobs: Serialize span context into job parameters

// Producer: Inject context into Kafka message
const span = tracer.startSpan('kafka.produce');
const headers = {};
tracer.inject(span.context(), FORMAT_HTTP_HEADERS, headers);

await producer.send({
  topic: 'orders',
  messages: [{ value: orderData, headers }]
});

// Consumer: Extract context from Kafka message
const message = await consumer.consume();
const parentContext = tracer.extract(FORMAT_HTTP_HEADERS, message.headers);
const span = tracer.startSpan('kafka.consume', { childOf: parentContext });

4. Instrumentation

Auto-Instrumentation

Auto-instrumentation uses language agents to automatically inject tracing code into common libraries (HTTP clients, database drivers, frameworks). No code changes required:

Java Agent

java -javaagent:opentelemetry-javaagent.jar \
     -Dotel.service.name=order-service \
     -Dotel.exporter.otlp.endpoint=http://collector:4318 \
     -jar app.jar

Automatically instruments: Spring Boot, JDBC, Hibernate, Kafka, Redis, HTTP clients, etc.

Python Auto-Instrumentation

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument python app.py

Automatically instruments: Flask, Django, requests, psycopg2, pymongo, etc.

Manual Instrumentation

For custom logic, business operations, or languages without auto-instrumentation:

// Node.js with OpenTelemetry
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('payment-service');

async function processPayment(orderId, amount) {
  const span = tracer.startSpan('payment.process', {
    attributes: {
      'order.id': orderId,
      'payment.amount': amount,
      'payment.currency': 'USD'
    }
  });

  try {
    // Business logic
    const result = await chargeCard(amount);

    span.setAttribute('payment.transaction_id', result.txnId);
    span.setStatus({ code: SpanStatusCode.OK });

    return result;
  } catch (error) {
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message
    });
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

Library Instrumentation

Third-party libraries can provide built-in tracing support:

// Express.js middleware
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');

registerInstrumentations({
  instrumentations: [
    new ExpressInstrumentation({
      requestHook: (span, requestInfo) => {
        span.setAttribute('user.id', requestInfo.request.user?.id);
      }
    })
  ]
});

// Automatic span creation for all routes
app.get('/orders/:id', async (req, res) => {
  // Span automatically created and propagated
  const order = await db.query('SELECT * FROM orders WHERE id = ?', [req.params.id]);
  res.json(order);
});

OpenTelemetry SDK

OpenTelemetry is the industry-standard observability framework. It provides:

• Vendor-neutral API: Write once, export to any backend (Jaeger, Zipkin, Tempo, Datadog, New Relic)
• Auto-instrumentation: For popular frameworks/libraries
• Context propagation: W3C TraceContext support
• Multiple signals: Traces, metrics, logs (unified context)
• Collector: Centralized agent for batching, sampling, routing

Instrumentation Best Practices

5. Sampling Strategies

Why Sampling?

At scale, tracing every request is expensive:

• Volume: 10,000 req/sec × 20 spans/req × 2 KB/span = 400 MB/sec = 34 TB/day
• Storage cost: Storing full traces for 30 days costs $50,000+/month
• Processing overhead: Network, serialization, backend indexing
• Query performance: Billions of spans slow down search

Sampling reduces cost by recording only a fraction of traces. The tradeoff: lower resolution but still sufficient for debugging and analysis.

Head-Based Sampling

Head-based sampling makes the sampling decision at trace creation (the "head" of the request). Once decided, the decision propagates to all services via trace_flags.

Probabilistic Sampling

Sample X% of traces randomly:

// Sample 1% of traces
const sampler = new TraceIdRatioBasedSampler(0.01);

const provider = new TracerProvider({
  sampler: sampler
});

Pros: Simple, predictable cost, unbiased sample
Cons: Rare errors might be missed, no adaptivity

Rate-Limiting Sampling

Sample up to N traces per second:

const sampler = new RateLimitingSampler(100); // Max 100 traces/sec

Pros: Predictable backend load
Cons: Biased toward low-traffic periods

Tail-Based Sampling

Tail-based sampling makes the decision after the trace completes. The backend buffers traces briefly, then decides which to keep based on criteria:

• Error-based: Keep all traces with errors
• Latency-based: Keep slow traces (p99, p999)
• Attribute-based: Keep traces with specific user IDs, endpoints
• Probabilistic fallback: Sample 1% of normal traces

Pros: Keep all interesting traces (errors, slow requests), adaptive to traffic patterns
Cons: Requires buffering (memory/latency), needs centralized collector, complex implementation

Adaptive Sampling

Adaptive sampling adjusts sample rate based on system conditions:

• Error rate: Increase sampling when error rate spikes
• Latency: Increase sampling during latency incidents
• Traffic volume: Decrease sampling during traffic spikes
• Cost budget: Adjust to stay within budget limits

// Pseudo-code for adaptive sampling
if (errorRate > 0.05) {
  sampleRate = 0.10; // Sample 10% during incidents
} else if (p99Latency > SLA) {
  sampleRate = 0.05; // Sample 5% during slowdowns
} else {
  sampleRate = 0.01; // Sample 1% normally
}

Sampling Strategy Comparison

Recommended Strategy

For production systems, use a hybrid approach:

1. Head-based: Sample 1-5% of traces probabilistically
2. Tail-based: Always keep traces with:
   • HTTP 4xx/5xx status codes
   • Latency > p99 threshold
   • Spans with error flag set
3. Adaptive: Increase head-based sampling during incidents (error spike, latency spike)

This ensures you always have error/slow traces while keeping costs predictable.

6. Storage & Backends

Storage Requirements

Tracing backends must handle:

• High write throughput: Thousands of spans/sec
• Time-series data: Spans arrive in trace order
• Fast queries: Find trace by ID in milliseconds
• Rich filtering: Query by service, operation, latency, attributes
• Retention: Store 7-30 days, then delete

Jaeger

Jaeger is the most popular open-source tracing backend (originally from Uber).

Architecture

• Agent: Sidecar that batches spans locally, reduces network calls
• Collector: Receives spans, writes to storage
• Storage: Cassandra (scalable) or Elasticsearch (rich queries)
• Query: API for retrieving traces
• UI: Web interface with waterfall diagrams, service maps

Pros: Mature, feature-rich UI, Cassandra scales to billions of spans
Cons: Operational complexity (run Cassandra/ES cluster), expensive storage, indexing overhead

Tempo

Grafana Tempo is a cost-efficient tracing backend designed for high volume.

Architecture

• Object storage: Stores traces as Parquet files in S3/GCS (cheap)
• No indexing: Traces are only queryable by trace_id (not by attributes)
• Bloom filters: Efficiently find traces without full indexing
• Integration: Native Grafana integration, correlate with logs/metrics

Pros: 10x cheaper storage, simple ops (no database), scales to trillions of spans
Cons: Limited search (trace_id only), no attribute-based queries, must integrate with logs/metrics for discovery

Zipkin

Zipkin is the original distributed tracing system (from Twitter, 2012).

• Storage: MySQL, Elasticsearch, or in-memory
• Simpler: Single binary, easier to run than Jaeger
• Less scalable: MySQL struggles at high volume
• Legacy: Fewer features than Jaeger/Tempo

Use case: Small deployments, dev/test environments, quick setup

Backend Comparison

Cloud Provider Options

• AWS X-Ray: Managed tracing, integrates with Lambda/ECS/EKS, basic UI
• Google Cloud Trace: Managed, integrates with GKE/Cloud Run, good UI
• Azure Application Insights: Full observability platform, rich correlation

These are convenient but more expensive than self-hosted and lock you into the vendor.

7. Trace Analysis

Finding Bottlenecks

The primary use of traces is identifying performance bottlenecks. Key techniques:

1. Waterfall Inspection

Look for long spans in the critical path:

Trace ID: abc123 | Total: 3.2s

GET /checkout                           |====================================| 3.2s
  auth.verify                            |=|                                     0.1s
  order.create                              |==============================| 2.8s
    inventory.check                              |===|                         0.3s
    payment.charge                               |====================|      2.0s ← BOTTLENECK
      stripe.api.call                                |=================|     1.8s ← ROOT CAUSE
    email.send                                                     |=|       0.2s

Analysis: 56% of latency is Stripe API. Optimize: cache payment tokens, parallel processing.
                    

2. Span Duration Aggregation

Query for slow operations across all traces:

SELECT operation, AVG(duration), P50(duration), P99(duration), COUNT(*)
FROM spans
WHERE timestamp > NOW() - 1h
GROUP BY operation
ORDER BY P99(duration) DESC;

Results:
stripe.api.call       avg=1.8s  p99=5.2s   count=12,450  ← Investigate
db.query.orders       avg=0.05s p99=0.8s   count=45,230
redis.get             avg=0.003s p99=0.01s count=230,450

3. Error Correlation

Find traces with errors and analyze common patterns:

// Find all traces with errors
WHERE status = ERROR

// Group by error type
GROUP BY error_message

Results:
"Connection timeout to payment-service"  count=1,234  ← 500s at checkout
"Invalid inventory ID"                   count=45     ← Data issue

Critical Path Analysis

The critical path is the longest execution path in the trace. Optimizing non-critical spans doesn't improve latency:

                     ┌─ inventory.check (0.3s, parallel)
order.create (2.8s) ─┼─ payment.charge (2.0s, critical path)
                     └─ email.send (0.2s, parallel)

Critical path: order.create → payment.charge = 2.0s
Optimizing inventory.check saves 0s (it's parallel with payment.charge).
Optimizing payment.charge saves up to 2.0s (it's blocking).
                    

Service Dependency Maps

Traces reveal service dependencies automatically:

Error Correlation

Traces make it easy to correlate errors across services:

• Root cause: Find the first error in the trace (earliest span with error flag)
• Cascading failures: See how upstream errors cause downstream timeouts/errors
• Partial failures: Identify which services succeeded vs failed in a multi-step flow

Latency Regression Detection

Compare traces before/after a deployment to detect regressions:

// Before deployment
SELECT P50(duration), P99(duration)
FROM traces
WHERE service = 'order-service'
  AND timestamp BETWEEN deploy_time - 1h AND deploy_time;

// After deployment
SELECT P50(duration), P99(duration)
FROM traces
WHERE service = 'order-service'
  AND timestamp BETWEEN deploy_time AND deploy_time + 1h;

Result:
Before: p50=0.5s, p99=1.2s
After:  p50=0.9s, p99=2.8s  ← Regression! Rollback or investigate.

Trace Comparison

Compare two traces side-by-side to find differences:

Insight: payment.charge is 10x slower. Check span attributes for details (retries? timeout? external API latency?).

8. Performance Impact

Overhead of Tracing

Distributed tracing adds overhead to every request:

• CPU: Span creation, serialization (Protobuf/JSON), context propagation
• Memory: Buffering spans before export
• Network: Sending spans to collector
• Latency: If blocking on export (rare)

Typical overhead: 1-5% of application CPU and latency. At high traffic (10K+ req/sec), this can be significant.

Reducing Cost

1. Sampling

Most effective: Sample 1-5% instead of 100%.

• Cost reduction: 20-100x
• Tradeoff: Lower resolution, rare events missed
• Mitigation: Tail-based sampling keeps errors/slow traces

2. Async Export

Export spans in background threads, don't block requests:

// Bad: Blocking export adds latency
span.end();
exporter.export(span); // Blocks request for 10-50ms

// Good: Async export
span.end();
spanBuffer.add(span); // Non-blocking
// Background thread flushes buffer periodically

OpenTelemetry SDK does this by default.

3. Batching

Send spans in batches (100-1000 at a time) instead of individually:

• Network efficiency: 1 request instead of 1000
• Backend efficiency: Bulk writes faster than individual writes
• Tradeoff: Slight export delay (1-5 seconds)

const exporter = new OTLPTraceExporter({
  url: 'http://collector:4318/v1/traces',
  maxQueueSize: 2048,
  maxExportBatchSize: 512,
  scheduledDelayMillis: 5000 // Flush every 5s
});

4. Reduce Span Cardinality

Avoid high-cardinality attributes that explode storage:

// Bad: Unbounded cardinality
span.setAttribute('user.email', 'user@example.com'); // Millions of unique values

// Good: Low cardinality
span.setAttribute('user.tier', 'premium'); // Few unique values
// Store email in logs, not spans

5. Compression

Compress spans before sending (gzip, zstd):

const exporter = new OTLPTraceExporter({
  compression: 'gzip' // 70-90% size reduction
});

Cost Analysis

Example: 10,000 req/sec, 20 spans/req, 100% sampling:

Performance Testing

Before deploying tracing to production:

1. Load test: Measure latency/throughput with tracing on vs off
2. Resource usage: Monitor CPU, memory, network with tracing enabled
3. Backend capacity: Ensure collector/backend can handle peak load
4. Export failure: Verify app continues working if collector is down

// Load test results
Without tracing: p99 latency = 450ms, CPU = 60%
With tracing:    p99 latency = 475ms, CPU = 62%

Overhead: +25ms latency (+5.5%), +2% CPU
Acceptable for most systems.

9. Service Maps

Auto-Generated Dependency Graphs

One of tracing's killer features: automatically generate service dependency graphs from traces. No manual documentation needed.

How It Works

The tracing backend analyzes traces and builds a graph:

1. Each unique service.name becomes a node
2. Parent-child span relationships become edges
3. Aggregate metrics (request rate, latency, error rate) annotate edges

Traffic Flow Visualization

Service maps show request flow and volume:

• Edge width: Proportional to request rate
• Edge color: Green (healthy), yellow (slow), red (errors)
• Node size: Request volume or latency
• Latency heatmap: Which services are slow?

Use Cases

1. Onboarding New Engineers

Service maps are live architecture diagrams. New engineers can see:

• What services exist
• How they interact
• Which services are critical (high traffic)
• Which are flaky (high error rate)

2. Impact Analysis

Before modifying a service, check its dependencies:

• Which services call this one? (Upstream dependencies)
• Which services does it call? (Downstream dependencies)
• What's the blast radius if it goes down?

3. Incident Response

During incidents, service maps show:

• Blast radius: Which services are affected?
• Root cause: Where did errors start?
• Cascading failures: How did errors propagate?

Incident: Payment Service down (100% errors)

Service Map Analysis:
- Upstream: Order Service (8K req/s → 100% timeout errors)
- Downstream: Stripe API (0 req/s, Payment Service not reaching it)
- Blast radius: Order Service, Checkout flow, 8K users/sec affected

Root cause: Payment Service pod crash-looping (OOM).
Fix: Scale up memory, redeploy.
                    

4. Architecture Smells

Service maps reveal architecture issues:

• Chatty services: Service A calls Service B 100 times per request (N+1 queries)
• Circular dependencies: A → B → C → A (deadlock risk)
• God services: One service called by all others (SPOF, bottleneck)
• Orphaned services: No incoming traffic (unused, safe to delete?)

Real-Time vs Historical

• Real-time: Service map updates continuously, shows current state. Use for monitoring, incident response.
• Historical: Generate service map for a specific time window. Use for postmortems, capacity planning, comparing before/after changes.

Multi-Cluster Service Maps

In multi-region/multi-cluster deployments, service maps can show cross-region traffic:

10. Best Practices

Span Naming Conventions

Use consistent, hierarchical span names for aggregation and filtering:

Attribute Standards

Use OpenTelemetry semantic conventions for common attributes:

HTTP Attributes

{
  "http.method": "POST",
  "http.route": "/api/orders",
  "http.status_code": 201,
  "http.url": "https://api.example.com/api/orders",
  "http.user_agent": "Mozilla/5.0...",
  "http.request_content_length": 1024,
  "http.response_content_length": 256
}

Database Attributes

{
  "db.system": "postgresql",
  "db.name": "production",
  "db.statement": "SELECT * FROM orders WHERE id = ?",
  "db.operation": "SELECT",
  "db.connection_string": "postgresql://localhost:5432"
}

Custom Business Attributes

{
  "user.id": "usr_abc123",
  "user.tier": "premium",
  "tenant.id": "tenant_xyz",
  "order.id": "ord_456",
  "order.total": 49.99,
  "payment.gateway": "stripe",
  "feature.flag.checkout_v2": true
}

Trace Context in Async Operations

Propagating context through queues and background jobs:

Message Queue (Kafka)

// Producer: Inject context
const span = tracer.startSpan('kafka.produce');
const headers = {};
propagation.inject(context.active(), headers);

await producer.send({
  topic: 'order-created',
  messages: [{
    key: orderId,
    value: orderData,
    headers: headers // Propagate trace context
  }]
});
span.end();

// Consumer: Extract context
for await (const message of consumer) {
  const parentContext = propagation.extract(context.active(), message.headers);
  const span = tracer.startSpan('kafka.consume', { parent: parentContext });

  await processOrder(message.value);
  span.end();
}

Background Jobs (Redis Queue)

// Enqueue job with trace context
const span = tracer.startSpan('job.enqueue');
const traceContext = {};
propagation.inject(context.active(), traceContext);

await queue.add('process-order', {
  orderId: '12345',
  _trace: traceContext // Embed trace context in job data
});
span.end();

// Worker: Extract and continue trace
queue.process('process-order', async (job) => {
  const parentContext = propagation.extract(context.active(), job.data._trace);
  const span = tracer.startSpan('job.process', { parent: parentContext });

  await processOrder(job.data.orderId);
  span.end();
});

Baggage for Cross-Cutting Data

Use baggage to propagate context that all services need:

// API Gateway: Set baggage
const span = tracer.startSpan('http.request');
baggage.setItem('user.id', userId);
baggage.setItem('tenant.id', tenantId);
baggage.setItem('request.id', requestId);

// All downstream services automatically get baggage
function processOrder() {
  const userId = baggage.getItem('user.id'); // Available without passing parameters
  const span = tracer.startSpan('order.create', {
    attributes: {
      'user.id': userId // Add to span for searchability
    }
  });
}

Use cases:

• User/tenant ID (for multi-tenancy, filtering)
• Request ID (correlation with logs)
• Feature flags (A/B testing, canary)
• Security context (auth scopes, roles)

Span Lifecycle Management

Always close spans, even on errors:

// Bad: Span leaks on error
const span = tracer.startSpan('operation');
await doWork(); // Throws error
span.end(); // Never called

// Good: Use try-finally
const span = tracer.startSpan('operation');
try {
  await doWork();
  span.setStatus({ code: SpanStatusCode.OK });
} catch (error) {
  span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
  span.recordException(error);
  throw error;
} finally {
  span.end(); // Always called
}

Testing Strategies

1. Unit Tests with Mock Tracer

const { InMemorySpanExporter, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');

const exporter = new InMemorySpanExporter();
const provider = new TracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

// Run code under test
await processOrder('12345');

// Assert spans
const spans = exporter.getFinishedSpans();
expect(spans).toHaveLength(3);
expect(spans[0].name).toBe('order.create');
expect(spans[0].attributes['order.id']).toBe('12345');

2. Integration Tests: Verify Context Propagation

// Test that trace_id propagates across services
const traceId = '4bf92f3577b34da6a3ce929d0e0e4736';

// Service A
const response = await fetch('http://service-a', {
  headers: { 'traceparent': `00-${traceId}-0000000000000001-01` }
});

// Service B (called by Service A)
const serviceBSpans = await getSpans('service-b', traceId);
expect(serviceBSpans[0].traceId).toBe(traceId); // Same trace!

Production Checklist