Chapter 7: Reliability & Fault Tolerance

MTBF, MTTR, MTTF

• MTBF (Mean Time Between Failures): Average time between failures for repairable systems. If a server fails every 720 hours on average, MTBF = 720 hours = 30 days.
• MTTR (Mean Time To Repair): Average time to restore service after failure. If average incident takes 2 hours to fix, MTTR = 2 hours.
• MTTF (Mean Time To Failure): Average time to failure for non-repairable components (e.g., hard drives). Once failed, you replace them.

graph LR
    A[System Running] -->|MTTF or MTBF| B[Failure Occurs]
    B -->|Detection Time| C[Failure Detected]
    C -->|MTTR| D[System Restored]
    D -->|MTBF| B

    style A fill:#2d5016
    style B fill:#8b0000
    style D fill:#2d5016

graph TD
    SLI[SLI: Service Level Indicator
Actual measurement
e.g., Request success rate = 99.95%]
    SLO[SLO: Service Level Objective
Internal target
e.g., Success rate >= 99.9%]
    SLA[SLA: Service Level Agreement
Contractual commitment
e.g., Success rate >= 99.5% or refund]

    SLI -->|Measured against| SLO
    SLO -->|Buffers| SLA

    style SLI fill:#1a5490
    style SLO fill:#4a7ba7
    style SLA fill:#2d5016

Definitions

• SLI (Service Level Indicator): A quantitative measure of service level. Examples: request latency (p99 < 200ms), error rate (< 0.1%), throughput (>= 1000 QPS).
• SLO (Service Level Objective): Internal target for an SLI. Sets the bar for reliability. If you miss SLOs, you slow down feature work to improve reliability.
• SLA (Service Level Agreement): Contractual promise to customers, with consequences (refunds, credits) if violated. Always set SLAs more lenient than SLOs to give yourself error budget.

The Core Idea

If your SLO is 99.9% availability, you have a 0.1% error budget per month (43.8 minutes of downtime). This budget can be spent on:

• Planned maintenance
• Risky deployments
• Infrastructure changes
• Experiments

Policy: If you exhaust your error budget, you freeze feature releases and focus 100% on reliability improvements until the budget replenishes next month.

graph LR
    A[Error Budget Healthy
99.95% actual vs 99.9% SLO]
    B[Error Budget Exhausted
99.88% actual vs 99.9% SLO]

    A -->|Ship features
Take risks| A
    A -->|Incidents| B
    B -->|Freeze releases
Focus on reliability| C[Budget Replenishes]
    C --> A

    style A fill:#2d5016
    style B fill:#8b0000
    style C fill:#4a7ba7

graph TD
    F[Failure Types]
    F --> C[Crash Failures
Process dies]
    F --> O[Omission Failures
Message lost]
    F --> T[Timing Failures
Too slow]
    F --> B[Byzantine Failures
Arbitrary behavior]
    F --> G[Gray Failures
Partial degradation]
    F --> CA[Cascading Failures
One triggers many]

    style C fill:#8b0000
    style O fill:#8b4500
    style T fill:#b8860b
    style B fill:#4b0082
    style G fill:#2f4f4f
    style CA fill:#800020

What It Is

Process dies suddenly (OOM kill, segfault, unhandled exception, power loss). Clean failure: the process stops responding entirely.

Detection

• Health check timeout
• TCP connection refused
• Process monitoring (systemd, Kubernetes liveness probe)

Mitigation

• Automatic restarts (systemd, Kubernetes ReplicaSet)
• Load balancer removes crashed instance from rotation
• Write-ahead logging (WAL) for state recovery

What It Is

Request or response is lost in transit. Network drops the packet, congestion causes buffer overflow, firewall rule blocks it.

Detection

• Client timeout (no response received)
• TCP retransmission failures (RST packet)
• Application-level ACKs missing

Mitigation

• TCP ensures reliable delivery (automatic retransmission)
• Application-level retries with exponential backoff
• Idempotency to allow safe retries
• Message queues with at-least-once delivery

sequenceDiagram
    participant C as Client
    participant N as Network
    participant S as Server

    C->>N: Request
    N-xS: Packet dropped!
    Note over C: Timeout after 5s
    C->>N: Retry request
    N->>S: Request delivered
    S->>N: Response
    N->>C: Response

What It Is

Response arrives, but too late. Service is slow due to overload, GC pause, blocking I/O, or resource contention.

Why It's Dangerous

Client times out and retries, but original request is still processing. Now two requests are in flight, doubling load. This can cascade into retry storms.

Detection

• Latency percentiles (p99, p99.9) exceed thresholds
• Client timeout counters spike
• Slow query logs

Mitigation

• Deadline propagation (pass remaining time budget to downstream services)
• Cancel slow requests (don't waste resources)
• Rate limiting to prevent overload
• Circuit breaker to stop hitting slow dependencies

What It Is

Arbitrary or malicious behavior. Process returns wrong results, corrupts data, sends conflicting messages to different peers, or behaves unpredictably.

Causes

• Software bugs (e.g., integer overflow, race condition)
• Bit flips from cosmic rays (rare but real)
• Malicious actors (compromised node)
• Hardware failures (bad RAM, disk corruption)

Detection

• Checksums and cryptographic signatures
• Consensus protocols (Raft, Paxos) detect conflicting votes
• End-to-end validation (don't trust intermediate nodes)

Mitigation

• Byzantine Fault Tolerant (BFT) consensus (requires 3f+1 nodes to tolerate f failures)
• Input validation and output verification
• ECC memory, disk scrubbing
• Immutable logs for forensics

What It Is

The most insidious failure mode. System is partially degraded but not fully failed. Examples:

• 99% of requests succeed, 1% hang for 30 seconds
• Service responds to health checks but returns errors for real traffic
• Disk is slow (high latency) but not dead
• Network drops 5% of packets (enough to slow TCP but not fail)

Why It's Hard

Binary health checks (alive/dead) don't detect it. The system looks healthy to monitoring but users experience failures.

Detection

• Canary requests (send synthetic production-like traffic to test real paths)
• Client-side metrics (measure success rate from caller's perspective)
• Distributed tracing (spot slow spans)
• Outlier detection (one instance slower than peers)

Mitigation

• Adaptive health checks (mark instance unhealthy if error rate or latency exceeds threshold, even if process is alive)
• Circuit breaker (stop sending traffic to degraded instance)
• Graceful restart (kill and restart instances showing symptoms)

graph LR
    LB[Load Balancer]
    A[Instance A
Healthy
p99=50ms]
    B[Instance B
GRAY FAILURE
p99=2000ms]
    C[Instance C
Healthy
p99=50ms]

    LB --> A
    LB -.->|Health check: OK
But 1% of requests hang| B
    LB --> C

    style A fill:#2d5016
    style B fill:#8b4500
    style C fill:#2d5016

What It Is

One failure triggers others in a domino effect. Classic scenario:

1. One instance crashes
2. Load balancer shifts traffic to remaining instances
3. Increased load causes them to slow down
4. Clients timeout and retry, adding more load
5. Remaining instances crash from overload
6. Entire service collapses

Mitigation

• Capacity planning (N+2 redundancy: system survives losing 2 instances)
• Load shedding (reject new requests if overloaded)
• Circuit breaker (stop sending traffic to failing service)
• Backpressure (slow down upstream instead of crashing)
• Bulkheads (isolate failure domains)

sequenceDiagram
    participant C as Client
    participant LB as Load Balancer
    participant S1 as Server 1
    participant S2 as Server 2
    participant S3 as Server 3

    Note over S1: Crash!
    C->>LB: Request
    LB->>S2: Route (now 2x load)
    C->>LB: Request
    LB->>S3: Route (now 2x load)
    Note over S2,S3: Overloaded, slow responses
    Note over C: Timeout, retry
    C->>LB: Retry
    C->>LB: Retry
    Note over S2: Crash from overload!
    Note over S3: Crash from overload!
    Note over LB: All instances dead

How It Works

All nodes actively serve traffic simultaneously. Load balancer distributes requests across all instances.

Pros

• Maximum resource utilization (no idle standbys)
• Instant failover (just stop routing to failed instance)
• Horizontal scalability (add more instances to handle more load)

Cons

• Requires stateless services or distributed state management
• More complex (need coordination for stateful workloads)
• Cost (running multiple instances at all times)

Use When

• Service is stateless (web servers, API gateways)
• State is externalized (session in Redis, data in DB)
• Need maximum availability and performance

graph TD
    LB[Load Balancer]
    A1[Instance 1
ACTIVE
Serving 33% traffic]
    A2[Instance 2
ACTIVE
Serving 33% traffic]
    A3[Instance 3
ACTIVE
Serving 33% traffic]

    LB -->|33%| A1
    LB -->|33%| A2
    LB -->|34%| A3

    style A1 fill:#2d5016
    style A2 fill:#2d5016
    style A3 fill:#2d5016

How It Works

One node (primary) serves all traffic. One or more standby nodes are idle, waiting to take over if primary fails.

Pros

• Simpler state management (only one writer at a time)
• Works well for stateful services (databases, message queues)
• No split-brain risk during normal operation

Cons

• Wasted capacity (standbys are idle)
• Failover delay (detection + promotion time)
• Risk of data loss if primary dies before replicating state

Use When

• Service is stateful and hard to distribute (traditional RDBMS)
• Strong consistency required (single writer avoids conflicts)
• Cost-sensitive (fewer active instances)

graph TD
    LB[Load Balancer]
    P[Primary
ACTIVE
Serving 100% traffic]
    S1[Standby 1
PASSIVE
Replicating]
    S2[Standby 2
PASSIVE
Replicating]

    LB -->|100%| P
    P -.->|Replication| S1
    P -.->|Replication| S2

    style P fill:#2d5016
    style S1 fill:#4a7ba7
    style S2 fill:#4a7ba7

sequenceDiagram
    participant C as Client
    participant LB as Load Balancer
    participant P as Primary
    participant S as Standby

    C->>LB: Request
    LB->>P: Forward
    P->>LB: Response
    LB->>C: Response
    P->>S: Replicate state

    Note over P: Crash!
    Note over LB: Detect failure (30s)
    Note over LB: Promote standby
    Note over S: Now primary

    C->>LB: Request
    LB->>S: Forward
    S->>LB: Response

N+1 Redundancy

If you need N instances to handle load, run N+1. System survives losing 1 instance.

Example: Need 10 instances for peak load → run 11. If 1 crashes, remaining 10 handle load.

Problem: What if you lose 2? Or if 1 crashes during maintenance on another?

N+2 Redundancy

Run N+2 instances. System survives losing 2 instances simultaneously.

Better for: Production systems where rolling deployments take one instance out of rotation. With N+2, you can do maintenance on 1 while still having N+1 for failures.

2N Redundancy

Run double the required capacity. System survives losing 50% of instances.

Use when: Can't tolerate any degradation even during mass failures (financial systems, medical devices).

Tradeoff: 2x cost.

graph LR
    subgraph "N+1: 4 instances for 3x load"
    A1[Instance]
    A2[Instance]
    A3[Instance]
    A4[+1 buffer]
    end

    subgraph "N+2: 5 instances for 3x load"
    B1[Instance]
    B2[Instance]
    B3[Instance]
    B4[+1 buffer]
    B5[+1 buffer]
    end

    subgraph "2N: 6 instances for 3x load"
    C1[Instance]
    C2[Instance]
    C3[Instance]
    C4[Spare]
    C5[Spare]
    C6[Spare]
    end

Full Copy Replication: Each replica stores a complete copy of data. Simple but storage-expensive.

• Synchronous replication: Write waits for all replicas to acknowledge. Strong consistency, higher latency.
• Asynchronous replication: Write returns before replicas acknowledge. Low latency, risk of data loss if primary fails before replicating.
• Semi-synchronous: Write waits for at least 1 replica (MySQL semi-sync). Balanced tradeoff.

Instead of storing N full copies, split data into M fragments and add K parity fragments. Can reconstruct original data from any M fragments. Survives losing up to K fragments.

Example: (8+4) erasure coding splits data into 8 fragments, adds 4 parity fragments. Total 12 fragments. Can lose any 4 and still reconstruct. Storage overhead: 1.5x (vs 3x for 3-way replication).

Tradeoff: Lower storage cost, but higher CPU cost (encoding/decoding) and slower recovery.

Use when: Cold data (archives, backups) where cost matters more than speed.

How It Works

DNS server returns IP of healthy instance. On failure, DNS updates record to point to standby. Clients re-resolve DNS and start hitting standby.

Pros

• Simple
• Works across regions/clouds
• No single point of failure (DNS itself is distributed)

Cons

• Slow TTL propagation: Even with low TTL (60s), clients and intermediaries cache DNS. Real failover can take 5-15 minutes.
• No session persistence (clients may land on different instance)
• Requires external health checking (DNS server must know instance is down)

Use When

• Cross-region failover (no shared load balancer)
• Can tolerate minutes of downtime
• Stateless services

sequenceDiagram
    participant C as Client
    participant DNS as DNS Server
    participant P as Primary (us-east-1)
    participant S as Standby (us-west-2)

    C->>DNS: Resolve api.example.com
    DNS->>C: 1.2.3.4 (Primary)
    C->>P: Request
    P->>C: Response

    Note over P: Crash!
    Note over DNS: Health check fails
Update record

    C->>DNS: Resolve api.example.com (cache expired)
    DNS->>C: 5.6.7.8 (Standby)
    C->>S: Request
    S->>C: Response

How It Works

Load balancer periodically probes each backend instance (e.g., every 5 seconds). If instance fails N consecutive checks (e.g., 3), mark it unhealthy and stop routing traffic. If it passes M consecutive checks (e.g., 2), mark healthy again.

Health Check Types

• TCP check: Try to open TCP connection. Fast but shallow (doesn't verify app is healthy).
• HTTP check: Send GET request, expect 200 OK. Better (checks app layer).
• Custom check: App returns health status based on internal checks (DB connectivity, disk space, etc.). Most thorough.

Configuration

• Interval: How often to check (e.g., 5s). Lower = faster detection, higher = less overhead.
• Timeout: How long to wait for response (e.g., 3s). Must be < interval.
• Unhealthy threshold: Consecutive failures before marking unhealthy (e.g., 3).
• Healthy threshold: Consecutive successes before marking healthy (e.g., 2).

Failover time: Interval × Unhealthy threshold = 5s × 3 = 15s

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Unhealthy: 3 consecutive failures
    Unhealthy --> Healthy: 2 consecutive successes

    Healthy: Instance receives traffic
    Unhealthy: Instance removed from rotation

HealthCheck:
  Target: HTTP:8080/health
  Interval: 5 seconds
  Timeout: 3 seconds
  UnhealthyThreshold: 3
  HealthyThreshold: 2

# Failover time: 15 seconds (5s × 3)

How It Works

Multiple instances share a Virtual IP address. Only one instance claims the VIP at a time. If primary fails, standby detects it (via heartbeat) and claims the VIP. Clients keep using the same IP, unaware of failover.

Protocols

• VRRP (Virtual Router Redundancy Protocol): Standard protocol for VIP failover. Primary sends heartbeats. If standbys stop receiving heartbeats, one promotes itself.
• Keepalived: Linux implementation of VRRP.

Pros

• Fast failover (1-3 seconds)
• Clients don't need to reconnect (same IP)
• Works at network layer (no app changes)

Cons

• Single VIP can't horizontally scale (only one active instance)
• Split-brain risk (two instances claiming VIP if network partitioned)
• Typically limited to same LAN/subnet

Use When

• Stateful services that can't scale horizontally (legacy apps)
• Need fast failover without DNS propagation delay
• Single-writer databases

graph TD
    C[Clients connect to VIP: 10.0.1.100]
    P[Primary
Owns VIP
Sends heartbeats]
    S[Standby
Monitors heartbeats]

    C -.-> |VIP routes to| P
    P -.->|Heartbeat every 1s| S

    style P fill:#2d5016
    style S fill:#4a7ba7

sequenceDiagram
    participant C as Client
    participant VIP as VIP 10.0.1.100
    participant P as Primary
    participant S as Standby

    C->>VIP: Request
    VIP->>P: Routed
    P->>VIP: Response
    VIP->>C: Response

    P->>S: Heartbeat
    P->>S: Heartbeat

    Note over P: Crash!
    Note over S: No heartbeat for 3s
    Note over S: Claim VIP

    C->>VIP: Request
    VIP->>S: Routed
    S->>VIP: Response

Automatic Failover

Orchestrator (e.g., MySQL Orchestrator, PostgreSQL Patroni, MongoDB Replica Set) monitors replicas. On primary failure:

1. Detect failure (timeout on health check or replication lag)
2. Elect new primary (most up-to-date replica)
3. Reconfigure replicas to follow new primary
4. Update VIP/DNS to point to new primary

Time: 30-120 seconds

Manual Failover

DBA manually promotes standby. Slower (minutes to hours) but safer (human validates state before promoting).

Split-Brain Problem

Network partition causes both primary and standby to believe they are primary. Both accept writes. When partition heals, data conflicts. Solution: Use quorum (require majority of replicas to agree before promoting). With 3 replicas, need 2 votes to become primary.

graph TD
    O[Orchestrator
Monitors replicas]
    P[Primary
Accepting writes]
    R1[Replica 1]
    R2[Replica 2]

    O -.->|Health check| P
    O -.->|Health check| R1
    O -.->|Health check| R2

    P -->|Replication| R1
    P -->|Replication| R2

    style P fill:#2d5016

sequenceDiagram
    participant O as Orchestrator
    participant P as Primary
    participant R1 as Replica 1
    participant R2 as Replica 2

    O->>P: Health check
    Note over P: Crash!
    O-xP: Health check timeout

    Note over O: Detect failure
Start election
    O->>R1: Get replication position
    O->>R2: Get replication position
    R1->>O: Position 1000
    R2->>O: Position 1005

    Note over O: R2 most up-to-date
Promote R2
    O->>R2: Promote to primary
    O->>R1: Follow R2

Best Practice: Externalize State

Turn stateful services into stateless ones by externalizing state:

• Store sessions in Redis/Memcached instead of in-process memory
• Store uploads in S3 instead of local disk
• Use distributed caches (Redis Cluster) instead of single-node caches

Now failover is trivial: just route to any healthy instance.

Your API depends on a downstream database. Database becomes slow (p99 latency goes from 10ms to 5000ms). Your API keeps calling it:

• Each request takes 5 seconds (timeout)
• Threads/connections are blocked waiting
• Request queue builds up
• Your API runs out of threads → starts rejecting requests
• Meanwhile, database is being hammered, can't recover

Insight: Continuing to call a failing dependency makes things worse for both you and the dependency.

Circuit breaker sits between you and dependency. Tracks success/failure rate. If failures exceed threshold, "opens" the circuit: stop calling dependency, fail fast immediately. After timeout, try again ("half-open"). If successful, close circuit and resume normal operation.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Failure rate > threshold
    Open --> HalfOpen: Timeout expires
    HalfOpen --> Closed: Success rate > threshold
    HalfOpen --> Open: Failure rate > threshold

    Closed: Normal operation
Allow all requests
Track success/failure
    Open: Fail fast
Reject all requests immediately
Don't call dependency
    HalfOpen: Trial period
Allow limited requests
Test if dependency recovered

Closed (Normal)

• Circuit is functioning normally
• All requests pass through to dependency
• Track metrics: success count, failure count, latency
• Transition to Open: If failure rate exceeds threshold (e.g., > 50% of last 100 requests failed)

Open (Failing Fast)

• Dependency is considered unavailable
• All requests fail immediately without calling dependency
• Return cached data, default value, or error
• Transition to Half-Open: After timeout (e.g., 60 seconds)

Why this helps:

• Protects your service (don't waste threads/connections on failing calls)
• Protects dependency (stop hammering it, give it time to recover)
• User experience (fail in 1ms instead of 5000ms)

Half-Open (Testing)

• Allow a small number of trial requests through (e.g., 10)
• If they succeed, dependency has recovered → Closed
• If they fail, dependency still broken → Open

sequenceDiagram
    participant A as API
    participant CB as Circuit Breaker
    participant DB as Database

    Note over CB: State: CLOSED
    A->>CB: Request
    CB->>DB: Forward
    DB->>CB: Response (slow, 5s)
    CB->>A: Response

    Note over CB: Failure count: 1/100

    loop 50 more slow responses
        A->>CB: Request
        CB->>DB: Forward
        DB->>CB: Slow response
        CB->>A: Response
    end

    Note over CB: Failure rate: 51/100 = 51%
Exceeds threshold
State: OPEN

    A->>CB: Request
    CB->>A: Fail fast (no DB call)

    Note over CB: Wait 60 seconds
    Note over CB: State: HALF-OPEN

    A->>CB: Request (trial)
    CB->>DB: Forward
    DB->>CB: Success (fast)
    CB->>A: Response

    Note over CB: 10/10 trial requests succeeded
State: CLOSED

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
  .failureRateThreshold(50)  // Open if 50% fail
  .minimumNumberOfCalls(20)   // Need 20 calls to calculate rate
  .waitDurationInOpenState(Duration.ofSeconds(60))
  .permittedNumberOfCallsInHalfOpenState(10)
  .slidingWindowSize(100)
  .slowCallDurationThreshold(Duration.ofSeconds(5))
  .slowCallRateThreshold(100)  // Open if all calls slow
  .build();

CircuitBreaker circuitBreaker = CircuitBreaker.of("database", config);

Supplier<String> decoratedSupplier = CircuitBreaker
  .decorateSupplier(circuitBreaker, () -> database.query());

Try<String> result = Try.ofSupplier(decoratedSupplier);

var circuitBreaker = Policy
  .Handle<Exception>()
  .CircuitBreakerAsync(
    exceptionsAllowedBeforeBreaking: 5,
    durationOfBreak: TimeSpan.FromSeconds(60),
    onBreak: (ex, duration) =>
    {
      Log.Warning("Circuit opened for {duration}", duration);
    },
    onReset: () => Log.Info("Circuit closed")
  );

var result = await circuitBreaker.ExecuteAsync(
  () => database.QueryAsync()
);

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
  Name:        "database",
  MaxRequests: 10,  // Half-open trial requests
  Interval:    time.Minute,  // Window size
  Timeout:     time.Minute,  // Open duration
  ReadyToTrip: func(counts gobreaker.Counts) bool {
    failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
    return counts.Requests >= 20 && failureRatio >= 0.5
  },
})

result, err := cb.Execute(func() (interface{}, error) {
  return database.Query()
})

When circuit is open, you can't call dependency. What do you return to user?

Option 1: Return Cached/Stale Data

If you have cached data, return it (mark as potentially stale). Better than error.

Example: Product catalog service can return yesterday's prices.

Option 2: Return Default/Fallback Value

Return a sensible default that allows operation to continue.

Example: Recommendations service returns "popular items" instead of personalized recs.

Option 3: Degrade Gracefully

Disable non-essential features, keep core functionality working.

Example: E-commerce checkout disables "recommended products" widget but allows purchase.

Option 4: Return Error

If no fallback is possible, return clear error. Better to fail fast (1ms) than slow (5000ms).

Retry Amplification

Scenario: Service is at 100% capacity. 1% of requests fail. Clients retry failed requests. Now service gets 101% load. More failures. More retries. System collapses under retry storm.

Example: Service handles 10,000 QPS. 1% fail (100 requests). Clients retry each once. Now 10,100 QPS. Service is overloaded. 5% fail (505 requests). Clients retry. Now 10,505 QPS. Failure rate hits 20%. System death spiral.

graph TD
    A[Service at 100% capacity
10,000 QPS]
    B[1% fail: 100 requests]
    C[Clients retry failed requests
Now 10,100 QPS]
    D[Service overloaded
5% fail: 505 requests]
    E[Clients retry
Now 10,505 QPS]
    F[Service collapses
80% fail]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F

    style A fill:#2d5016
    style D fill:#8b4500
    style F fill:#8b0000

Implementation

for (int i = 0; i < maxRetries; i++) {
  try {
    return service.call();
  } catch (Exception e) {
    if (i == maxRetries - 1) throw e;
  }
}

Problems

• No delay: Hammers service immediately, worsens overload
• Not idempotent-safe: If request partially succeeded (e.g., wrote to DB but response lost), retry causes duplicate write
• Thundering herd: All clients retry at same time

When to Use

• Never in production. Use for quick prototypes only.

How It Works

Wait before retrying. Double the wait time after each failure.

Example: Retry after 1s, then 2s, then 4s, then 8s, then 16s.

Implementation

int delay = 1000; // 1 second
for (int i = 0; i < maxRetries; i++) {
  try {
    return service.call();
  } catch (Exception e) {
    if (i == maxRetries - 1) throw e;
    Thread.sleep(delay);
    delay *= 2;
  }
}

Why It Helps

• Gives service time to recover
• Reduces retry rate as failures persist
• Eventually backs off enough to stop hammering

Problem

Thundering herd: If 1000 clients all fail at same time, they all retry after 1s, then 2s, then 4s. Still synchronized.

How It Works

Add randomness to delay. Instead of retrying after exactly 2s, retry after 1-2s (random). This desynchronizes clients.

Implementation

int baseDelay = 1000;
int delay = baseDelay;
Random random = new Random();

for (int i = 0; i < maxRetries; i++) {
  try {
    return service.call();
  } catch (Exception e) {
    if (i == maxRetries - 1) throw e;

    // Full jitter: random between 0 and delay
    int jitteredDelay = random.nextInt(delay);
    Thread.sleep(jitteredDelay);

    delay = Math.min(delay * 2, maxDelay);  // Cap at maxDelay
  }
}

Jitter Strategies

• Full jitter: random(0, delay) — Most aggressive desynchronization
• Equal jitter: delay/2 + random(0, delay/2) — Balances desync and average delay
• Decorrelated jitter: random(baseDelay, delay * 3) — AWS recommendation

graph LR
    A[1000 clients fail at t=0]
    B[Retry with exponential backoff
All retry at t=1s, t=2s, t=4s
Still synchronized]
    C[Retry with backoff + jitter
Retry spread across time
Desynchronized]

    A --> B
    A --> C

    style B fill:#8b4500
    style C fill:#2d5016

The Problem

Even with backoff and jitter, retries can amplify load. If service is truly overloaded, retries make it worse.

Solution: Limit % of Retries

Track: retry_ratio = retries / (requests + retries). If retry_ratio exceeds threshold (e.g., 20%), stop retrying. Only allow new requests.

Example: Service gets 10,000 requests. 1,000 fail. Clients want to retry all 1,000. But 1,000/(10,000+1,000) = 9% retry ratio. Allow it. But if 5,000 fail, 5,000/(10,000+5,000) = 33% retry ratio. Exceeds 20% budget. Only allow 3,000 retries. Drop the rest.

Why It Works

• Prevents retry amplification death spiral
• Maintains throughput during overload (prioritize new requests over retries)
• Automatically adapts to load

The Problem

If request is not idempotent, retrying can cause duplicates.

Example: POST /transfer?amount=100. Request succeeds (money transferred), but response is lost due to network error. Client retries. Now $200 transferred. User's bank account is wrong.

Solution 1: Idempotency Keys

Client generates unique ID for each request. Send it in header: Idempotency-Key: abc123. Server stores key after processing. If it sees same key again, returns cached response without re-executing.

POST /transfer HTTP/1.1
Idempotency-Key: abc123
{ "amount": 100 }

Server checks:
  - Is "abc123" in cache? Yes → return cached response
  - No → process request, cache result with key "abc123", return response

Solution 2: Only Retry Safe Methods

HTTP GET, HEAD, OPTIONS are safe (read-only, no side effects). Always safe to retry. POST, PUT, DELETE may have side effects. Only retry if idempotent.

graph TD
    A[Request Failed]
    B{Error Type}
    C{Is Request
Idempotent?}
    D[Retry with
Exp Backoff + Jitter]
    E[Don't Retry
Return Error]
    F{Retry Budget
Available?}
    G[Retry]
    H[Don't Retry
Fail Fast]

    A --> B
    B -->|4xx Client Error| E
    B -->|5xx Server Error| C
    B -->|Network Error| C
    C -->|Yes| F
    C -->|No| E
    F -->|Yes| G
    F -->|No| H

    G --> D

    style D fill:#2d5016
    style E fill:#8b0000
    style H fill:#8b4500

Your API has two endpoints:

• GET /critical — core business function
• GET /analytics — nice-to-have, slow, calls flaky third-party service

Both share the same thread pool (100 threads). Analytics endpoint gets hammered, becomes slow, ties up all 100 threads. Now critical endpoint can't get threads. Entire API is down because of one bad endpoint.

Give each endpoint its own thread pool:

• /critical: 80 threads
• /analytics: 20 threads

If analytics endpoint gets slow, it can only consume its 20 threads. Critical endpoint still has 80 threads available. Failure is contained.

graph TD
    A1["Shared Thread Pool 100 threads"]
    A1 --> A2
    A1 --> A3

    B1["Thread Pool 1 80 threads"]
    B2["Thread Pool 2 20 threads"]
    B1 --> B3
    B2 --> B4

    style B1 fill:#2d5016
    style B2 fill:#4a7ba7

Assign separate thread pools to different workloads.

Java Example: Dedicated ExecutorService

ExecutorService criticalPool = Executors.newFixedThreadPool(80);
ExecutorService analyticsPool = Executors.newFixedThreadPool(20);

// Critical endpoint
@GetMapping("/critical")
public CompletableFuture<String> critical() {
  return CompletableFuture.supplyAsync(() ->
    service.critical(),
    criticalPool
  );
}

// Analytics endpoint
@GetMapping("/analytics")
public CompletableFuture<String> analytics() {
  return CompletableFuture.supplyAsync(() ->
    service.analytics(),
    analyticsPool
  );
}

If you have two databases, give each its own connection pool. If DB1 becomes slow and exhausts its pool, DB2 connections are unaffected.

HikariConfig config1 = new HikariConfig();
config1.setMaximumPoolSize(50);
DataSource db1 = new HikariDataSource(config1);

HikariConfig config2 = new HikariConfig();
config2.setMaximumPoolSize(20);
DataSource db2 = new HikariDataSource(config2);

Run critical and non-critical workloads in separate processes. If analytics process crashes (OOM, segfault), critical process is unaffected.

Example: Microservices architecture. Each service is a separate process. One service crashing doesn't kill others.

Instead of separate thread pools, use semaphores to limit concurrent executions.

Semaphore criticalSemaphore = new Semaphore(80);
Semaphore analyticsSemaphore = new Semaphore(20);

@GetMapping("/critical")
public String critical() {
  criticalSemaphore.acquire();
  try {
    return service.critical();
  } finally {
    criticalSemaphore.release();
  }
}

@GetMapping("/analytics")
public String analytics() {
  if (!analyticsSemaphore.tryAcquire()) {
    throw new ServiceUnavailableException("Analytics overloaded");
  }
  try {
    return service.analytics();
  } finally {
    analyticsSemaphore.release();
  }
}

Advantage: Lighter weight than separate thread pools (no extra threads created).

Disadvantage: If analytics blocks all 20 permits, it blocks 20 threads from the shared pool (not isolated).

Considerations

• Criticality: Allocate more resources to critical endpoints
• Expected load: Size based on QPS and latency. If endpoint gets 100 QPS with 50ms latency, need ~5 threads (100 * 0.05 = 5)
• Failure blast radius: If endpoint calls flaky dependency, limit its pool size to prevent cascading failures
• Total capacity: Sum of all pools shouldn't exceed total threads (or you'll oversubscribe)

Example Allocation

• Total threads: 200
• Critical endpoints: 120 (60%)
• Important endpoints: 50 (25%)
• Non-critical endpoints: 20 (10%)
• Reserved for overhead: 10 (5%)

1. Connection Timeout

How long to wait when establishing TCP connection. If server doesn't accept connection within this time, fail.

Typical value: 2-5 seconds

Why shorter: Connection either succeeds quickly or won't succeed (server down, network unreachable). No point waiting long.

2. Read Timeout (Socket Timeout)

How long to wait for response after connection is established. If server doesn't send data within this time, fail.

Typical value: 10-30 seconds

Depends on: Expected latency of operation. Fast queries: 1s. Slow reports: 60s.

3. Write Timeout

How long to wait when sending request data to server. If can't send within this time, fail.

Typical value: 10 seconds

Rare to hit: Usually only if network is severely congested or server backpressure.

4. Request Timeout (End-to-End)

Total time from sending request to receiving response, including all retries. This is what user experiences.

Example: User clicks button. UI shows loading spinner. If no response after 30s, show error.

sequenceDiagram
    participant C as Client
    participant S as Server

    Note over C: Connection Timeout: 5s
    C->>S: SYN (start TCP handshake)
    S->>C: SYN-ACK
    C->>S: ACK
    Note over C: Connected in 0.1s

    Note over C: Write Timeout: 10s
    C->>S: Send HTTP request
    Note over C: Sent in 0.01s

    Note over C: Read Timeout: 30s
    Note over S: Processing... (20s)
    S->>C: Send HTTP response
    Note over C: Received in 20s

    Note over C: Total request time: 20.11s

RequestConfig config = RequestConfig.custom()
  .setConnectTimeout(5000)      // 5 seconds to establish connection
  .setSocketTimeout(30000)       // 30 seconds to read response
  .setConnectionRequestTimeout(2000)  // 2 seconds to get connection from pool
  .build();

CloseableHttpClient client = HttpClients.custom()
  .setDefaultRequestConfig(config)
  .build();

import requests

response = requests.get(
  'https://api.example.com/data',
  timeout=(5, 30)  # (connect timeout, read timeout)
)

client := &http.Client{
  Timeout: 30 * time.Second,  // Total request timeout
  Transport: &http.Transport{
    DialContext: (&net.Dialer{
      Timeout: 5 * time.Second,  // Connection timeout
    }).DialContext,
    ResponseHeaderTimeout: 10 * time.Second,  // Time to read headers
  },
}

The Problem with Fixed Timeouts

If you set timeout too low, you get false failures (requests that would've succeeded given more time). If too high, slow requests tie up resources for too long.

Solution: Adaptive Timeouts Based on Percentiles

Measure actual latency distribution. Set timeout at p99.9 (99.9th percentile). This allows 99.9% of requests to succeed while cutting off the slowest 0.1%.

Example: If p99.9 latency is 500ms, set timeout to 1000ms (2x p99.9 for safety margin). As latency changes over time, adjust timeout automatically.

Implementation

// Track latency in sliding window
LatencyTracker tracker = new LatencyTracker(windowSize = 10000);

// Update timeout periodically
scheduler.scheduleEvery(1.minute, () -> {
  double p999 = tracker.getPercentile(99.9);
  int newTimeout = (int)(p999 * 2);  // 2x p99.9
  httpClient.setTimeout(newTimeout);
  logger.info("Adjusted timeout to {}ms based on p99.9={}", newTimeout, p999);
});

The Problem

User makes request with 30s timeout. Your API calls Service A (20s), which calls Service B (15s), which calls Service C (10s). Total: 45s. Exceeds user's 30s deadline. Request fails, but all three services wasted resources processing it.

Solution: Propagate Deadline

Pass remaining time budget to downstream services. Each service checks: "Do I have enough time to fulfill this?" If not, reject immediately.

sequenceDiagram
    participant U as User
    participant A as API
    participant B as Service A
    participant C as Service B
    participant D as Service C

    U->>A: Request (deadline: 30s)
    Note over A: 2s elapsed
Remaining: 28s
    A->>B: Request (deadline: 28s)
    Note over B: 5s elapsed
Remaining: 23s
    B->>C: Request (deadline: 23s)
    Note over C: 8s elapsed
Remaining: 15s
    C->>D: Request (deadline: 15s)
    Note over D: Expected: 10s
Remaining: 15s
OK to proceed
    D->>C: Response (10s)
    C->>B: Response
    B->>A: Response
    A->>U: Response (total: 25s)

// Client sets deadline
ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 50051).build();
GreeterGrpc.GreeterBlockingStub stub = GreeterGrpc.newBlockingStub(channel);

HelloReply response = stub
  .withDeadlineAfter(30, TimeUnit.SECONDS)
  .sayHello(request);

// Server checks remaining time
if (Context.current().getDeadline().timeRemaining(TimeUnit.SECONDS) < 10) {
  throw new StatusException(Status.DEADLINE_EXCEEDED.withDescription("Not enough time"));
}

// DON'T
Socket socket = new Socket("example.com", 80);
socket.getInputStream().read();  // Blocks forever if no data

Perfect is the enemy of good: Users prefer a partially working system over a completely broken one. If your recommendation engine fails, don't fail checkout. If analytics database is down, don't block user signups.

Core Principle

Identify what's essential vs nice-to-have. When failures occur, drop nice-to-have features to keep essential ones running.

Use feature flags to selectively disable features under load or failure.

if (featureFlags.isEnabled("recommendations") && recommendationService.isHealthy()) {
  try {
    recommendations = recommendationService.get(userId);
  } catch (Exception e) {
    // Failed, but don't crash. Continue without recommendations.
    log.warn("Recommendations unavailable", e);
    metrics.increment("recommendations.fallback");
  }
}

// Render page with or without recommendations

Automatic Degradation

// Monitor error rate
if (errorRate > 10%) {
  featureFlags.disable("recommendations");
  log.alert("Auto-disabled recommendations due to high error rate");
}

// Re-enable when recovered
if (errorRate < 1% && timeSinceDisabled > 5.minutes) {
  featureFlags.enable("recommendations");
}

When primary data source fails, return fallback data.

Feature	Primary Source	Fallback
Product Recommendations	Personalized ML model	Most popular products
User Profile	User database	Anonymous user template
Pricing	Real-time pricing service	Cached prices (mark as stale)
Search	Elasticsearch cluster	Simple DB LIKE query (slow but works)
Shipping Estimate	Carrier API	Static estimate (3-5 days)

graph TD
    A[Request]
    B{Primary
Available?}
    C[Use Primary]
    D{Fallback
Available?}
    E[Use Fallback]
    F[Return Error]

    A --> B
    B -->|Yes| C
    B -->|No| D
    D -->|Yes| E
    D -->|No| F

    style C fill:#2d5016
    style E fill:#b8860b
    style F fill:#8b0000

Serve cached/stale data when fresh data unavailable. Inform user data may be outdated.

// Try primary database
try {
  data = database.query(key);
  cache.set(key, data, ttl=5.minutes);
  return data;
} catch (DatabaseException e) {
  // Database down, serve stale cache
  data = cache.get(key);
  if (data != null) {
    log.warn("Serving stale data for key={}", key);
    data.setStale(true);  // Mark as potentially outdated
    return data;
  }
  throw new ServiceUnavailableException("Data unavailable");
}

UI Indication

Show banner: "We're experiencing technical difficulties. Data shown may be up to 10 minutes old."

When writes are failing but reads work, enter read-only mode. Users can view data but not modify it.

Example: GitHub read-only mode during database incidents. Users can browse code, issues, PRs, but can't push commits or create issues.

if (database.isWriteUnhealthy()) {
  systemMode = READ_ONLY;
}

if (systemMode == READ_ONLY && request.method == POST|PUT|DELETE) {
  return 503 Service Unavailable {
    "error": "System is in read-only mode due to technical issues",
    "retry_after": 600
  }
}

Reduce quality to maintain throughput.

Service	Normal Quality	Degraded Quality
Video Streaming	1080p	720p or 480p
Image Processing	Full resolution	Thumbnails only
Search Results	100 results	10 results
ML Inference	Large model (slow, accurate)	Small model (fast, less accurate)

graph TD
    A[Classify Features]
    B[Critical
Checkout, Login, Payment]
    C[Important
Search, Product Pages]
    D[Nice-to-Have
Recommendations, Reviews]
    E[Non-Essential
Analytics, Ads, Trackers]

    A --> B
    A --> C
    A --> D
    A --> E

    B -.-> F[Never degrade
Always keep running]
    C -.-> G[Degrade last
Use fallbacks]
    D -.-> H[Degrade early
Drop if needed]
    E -.-> I[Degrade first
Drop immediately]

    style B fill:#8b0000
    style C fill:#b8860b
    style D fill:#4a7ba7
    style E fill:#2f4f4f

Server can handle 10,000 QPS with 50ms latency. At 12,000 QPS, it slows down to 500ms latency. At 15,000 QPS, it slows to 5000ms latency. All requests timeout. Effective throughput drops to 0.

Better approach: At 10,000 QPS, reject excess requests (return 503). Successfully serve 10,000 QPS. Better to serve 10k users well than fail 15k users completely.

graph LR
    subgraph "Without Load Shedding"
    A[15,000 QPS]
    B[All requests slow
5000ms latency]
    C[All timeout
0 successful]
    A --> B --> C
    end

    subgraph "With Load Shedding"
    D[15,000 QPS]
    E[Reject 5,000
Serve 10,000]
    F[10,000 successful
50ms latency]
    D --> E --> F
    end

    style C fill:#8b0000
    style F fill:#2d5016

Allow N requests per time window. Reject excess.

RateLimiter limiter = RateLimiter.create(10000);  // 10k QPS

if (limiter.tryAcquire()) {
  return service.handle(request);
} else {
  return 503 Service Unavailable {
    "error": "Rate limit exceeded",
    "retry_after": 1
  };
}

Problem: Bursty traffic can exceed limit within window. If limit is 10k/sec and 10k requests arrive in first 100ms, remaining 900ms has 0 capacity.

Bucket refills at fixed rate. Each request consumes token. If bucket empty, reject.

Advantage: Allows bursts (up to bucket capacity) while enforcing average rate.

Example: Bucket capacity 1000 tokens, refill rate 10,000/sec. Can handle bursts of 1000 requests, then throttles to 10k/sec.

// Guava RateLimiter uses token bucket
RateLimiter limiter = RateLimiter.create(10000.0);  // 10k tokens/sec
limiter.tryAcquire();  // Consumes 1 token

Not all requests are equal. Prioritize important requests, shed low-priority ones first.

Priority	Examples	Queue
Critical	Payment processing, checkout	Always served
High	Authenticated user traffic	Served if capacity available
Normal	Anonymous browse traffic	Shed if load high
Low	Bots, analytics, prefetch	Shed first

int priority = determinePriority(request);

if (priority == CRITICAL) {
  return service.handle(request);  // Always process
}

if (systemLoad > 90% && priority == LOW) {
  return 503;  // Shed low priority
}

if (systemLoad > 95% && priority == NORMAL) {
  return 503;  // Shed normal priority
}

return service.handle(request);

Measure actual system load (CPU, memory, queue depth). Reject requests when overloaded.

if (cpuUsage > 90% || queueDepth > 1000 || latencyP99 > 5000) {
  if (random() < 0.5) {  // Shed 50% of requests
    return 503 Service Unavailable;
  }
}

return service.handle(request);

Adaptive: Automatically adjusts rejection rate based on current load. Unlike fixed rate limits, responds to actual system health.

The Problem with Queue Depth

Traditional approach: reject when queue depth exceeds threshold. But what if queue is deep because you're processing quickly? You'd unnecessarily reject requests.

CoDel Solution

Track how long requests sit in queue (sojourn time). If queue latency exceeds threshold (e.g., 100ms) for a sustained period (e.g., 5s), start dropping requests.

Why better: Distinguishes between "long queue, but draining fast" (OK) vs "long queue, sitting forever" (overload).

// Simplified CoDel
long queueStartTime = now();
long queueEndTime = now();
long sojournTime = queueEndTime - queueStartTime;

if (sojournTime > 100ms) {
  overloadCounter++;
  if (overloadCounter > 5) {  // Sustained for 5 checks
    return 503;  // Start dropping
  }
} else {
  overloadCounter = 0;  // Reset counter
}

graph TD
    A[User]
    B[CDN/Edge]
    C[Load Balancer]
    D[API Gateway]
    E[Service]

    A --> B
    B --> C
    C --> D
    D --> E

    B -.-> F1[Shed: DDoS, bot traffic]
    C -.-> F2[Shed: Connection-level rate limiting]
    D -.-> F3[Shed: Request-level rate limiting,
priority queues]
    E -.-> F4[Shed: Admission control based on
actual system load]

Best Practice: Layer Defense

• Edge/CDN: Block obvious bad traffic (DDoS, known bots, geo-blocking)
• Load Balancer: Connection-level rate limits (max connections per IP)
• API Gateway: Request-level rate limits per user/API key
• Service: Admission control based on internal metrics (CPU, queue depth)

Each layer sheds what it can, protecting downstream layers.

When rejecting requests, tell client when to retry:

HTTP/1.1 503 Service Unavailable
Retry-After: 60

{
  "error": "Service overloaded",
  "message": "Please retry after 60 seconds"
}

Well-behaved clients will honor this and not hammer you with retries.

"It's better to break things yourself than have them break during a real incident."

Traditional testing: Test in staging, hope it works in prod. Problem: Staging never matches prod (different scale, different dependencies, different failure modes).

Chaos Engineering: Test in prod by injecting failures. Learn what actually breaks. Fix it before it causes real outages.

1. Define Steady State

Measure normal behavior. Example: p99 latency < 200ms, error rate < 0.1%, throughput >= 10k QPS.

2. Hypothesize Steady State Continues

"We believe that if we kill one instance, the system will continue serving requests with no user-visible impact."

3. Introduce Real-World Chaos

Inject failures that can happen in production: instance crashes, network partitions, disk full, high CPU, slow dependencies.

4. Run Experiments in Production

Staging can't replicate prod scale/complexity. Run chaos in prod (but minimize blast radius).

5. Automate Experiments

Run chaos continuously. Don't just test once and forget. Systems change; chaos must keep pace.

Kill random instances to verify redundancy and failover.

Netflix Chaos Monkey: Randomly kills EC2 instances during business hours. Forces teams to build resilient services that survive instance failures.

# Simulate instance crash
kubectl delete pod <random-pod>

# Verify: Does traffic automatically shift to healthy instances?
# Verify: Does load balancer detect failure and stop routing?
# Verify: Is a new instance automatically started?

Inject packet loss, latency, network partitions.

# Add 500ms latency to 10% of packets
tc qdisc add dev eth0 root netem delay 500ms 100ms 10%

# Drop 5% of packets
tc qdisc add dev eth0 root netem loss 5%

# Partition: block traffic between two services
iptables -A INPUT -s 10.0.1.0/24 -j DROP

Verify: Does system detect network issues? Do retries work? Does circuit breaker open?

Make dependencies slow or unavailable.

# Return errors from dependency
curl -X POST fault-injection-service/inject \
  -d '{
    "service": "database",
    "failure_rate": 0.5,
    "latency_ms": 5000
  }'

Verify: Does circuit breaker open? Does system use fallbacks? Is core functionality still working?

Consume CPU, memory, disk to simulate resource starvation.

# Stress CPU
stress-ng --cpu 4 --timeout 60s

# Fill disk
dd if=/dev/zero of=/tmp/bigfile bs=1M count=10000

# Memory leak simulation
while true; do malloc 100MB; sleep 1; done

Verify: Does autoscaling kick in? Do instances crash? Do they recover?

Clock skew, leap seconds, daylight saving time bugs.

# Set clock forward 1 hour
date -s "$(date -d '+1 hour')"

# Verify: Do certificates still validate? Do distributed locks break?

Netflix's suite of chaos tools:

• Chaos Monkey: Kills instances
• Chaos Kong: Kills entire AWS region
• Latency Monkey: Injects network latency
• Chaos Gorilla: Kills entire availability zone
• Doctor Monkey: Finds unhealthy instances
• Janitor Monkey: Cleans up unused resources
• Conformity Monkey: Finds instances that don't follow best practices

graph TD
    A[Chaos Engineering Tools]
    B[Chaos Monkey
Kill instances]
    C[Chaos Kong
Kill region]
    D[Latency Monkey
Slow network]
    E[Litmus
Kubernetes chaos]

    A --> B
    A --> C
    A --> D
    A --> E

    B -.-> F[Verify: Failover works]
    C -.-> G[Verify: Multi-region failover]
    D -.-> H[Verify: Timeouts, retries]
    E -.-> I[Verify: Pod recovery, health checks]

CNCF project for chaos engineering on Kubernetes.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
spec:
  appinfo:
    appns: default
    applabel: app=nginx
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      components:
        env:
        - name: TOTAL_CHAOS_DURATION
          value: '60'
        - name: CHAOS_INTERVAL
          value: '10'
        - name: FORCE
          value: 'false'

Continuously deletes pods every 10s for 60s. Verifies deployments, health checks, and autoscaling work.

1. Define Hypothesis

"If we kill 2 out of 10 API instances, remaining 8 will handle load with p99 latency < 300ms."

2. Define Blast Radius

Limit experiment scope to minimize risk. Example: Run on 10% of prod traffic. Have kill switch to abort.

3. Define Steady-State Metrics

• Error rate
• Latency (p50, p99)
• Throughput
• User-facing metric (successful checkouts, API calls)

4. Run Experiment

Inject failure. Monitor metrics in real-time. If metrics degrade beyond acceptable threshold, abort.

5. Analyze Results

• Did hypothesis hold? Why or why not?
• What broke? What worked?
• What improvements are needed?

6. Automate

Turn successful experiment into continuous test. Run weekly or daily.

Scheduled chaos engineering events where team simulates major outages.

Example Scenario

• 9:00 AM: Game day starts. Team on call.
• 9:15 AM: Facilitator kills primary database
• Team must: detect failure, promote replica, restore service
• Measure: Time to detect, time to recover, data loss
• 10:00 AM: Debrief. What went well? What failed? What to improve?

Why Valuable

• Tests runbooks and procedures
• Trains team for real incidents
• Finds gaps in monitoring and alerting
• Builds confidence

RPO (Recovery Point Objective)

How much data can you afford to lose?

• RPO = 0: Zero data loss (synchronous replication)
• RPO = 5 minutes: Lose up to 5 minutes of data (async replication with 5-min lag)
• RPO = 24 hours: Lose up to 1 day of data (daily backups)

RTO (Recovery Time Objective)

How quickly must you recover?

• RTO = 0: Instant failover (active-active)
• RTO = 5 minutes: Automatic failover to standby
• RTO = 4 hours: Manual failover, restore from backup
• RTO = 24 hours: Rebuild from scratch

graph LR
    A[Disaster Strikes]
    B[RPO: Last good backup]
    C[RTO: Time to restore]
    D[Service Restored]

    A --> B
    B --> C
    C --> D

    B -.-> E[Data loss window]
    C -.-> F[Downtime window]

    style E fill:#8b0000
    style F fill:#b8860b

The Tradeoff

Lower RPO/RTO = higher cost and complexity.

Target	Cost	Complexity
RPO=0, RTO=0 (active-active multi-region)	Very High (2x+ infrastructure)	Very High (data consistency, conflict resolution)
RPO=1min, RTO=5min (hot standby)	High	Medium
RPO=1hr, RTO=1hr (warm standby)	Medium	Medium
RPO=24hr, RTO=8hr (cold backup)	Low	Low

Choose based on business impact: How much revenue do you lose per hour of downtime? How critical is the data?

Periodically backup data to cold storage (S3 Glacier, tape). On disaster, restore from backup.

• RPO: Backup frequency (e.g., 24 hours for daily backups)
• RTO: Hours to days (depends on data size and restore speed)
• Cost: Very low (storage only)

Best For

Non-critical data, compliance/audit requirements, long-term archival.

Process

1. Provision new infrastructure (manual or IaC)
2. Install software
3. Restore data from backup
4. Verify integrity
5. Point DNS to new infrastructure

Minimal infrastructure always running in DR region (just database). On disaster, scale up remaining components.

• RPO: Minutes (database continuously replicated)
• RTO: 30 minutes to 1 hour (time to spin up app servers, scale database)
• Cost: Low (just database replicas running)

Best For

Business-critical apps with moderate downtime tolerance.

graph TD
    subgraph "Primary Region (us-east-1)"
    A1[App Servers
Running]
    A2[Database
Running]
    end

    subgraph "DR Region (us-west-2)"
    B1[App Servers
OFF]
    B2[Database Replica
Running]
    end

    A2 -.->|Continuous replication| B2

    C[Disaster!]
    C --> D[Scale up app servers
Promote DB replica
Update DNS]

    style A1 fill:#2d5016
    style A2 fill:#2d5016
    style B1 fill:#4a4a4a
    style B2 fill:#b8860b

Full infrastructure running in DR region, but at reduced capacity (e.g., 50% of production). On disaster, scale to full capacity.

• RPO: Minutes (continuous replication)
• RTO: 5-15 minutes (time to scale and switch DNS)
• Cost: Medium (50% infrastructure running)

Best For

Critical apps requiring fast recovery.

Full infrastructure running in multiple regions, all serving traffic simultaneously. No failover needed; remaining regions absorb load.

• RPO: 0 (synchronous replication or conflict-free data model)
• RTO: 0 (no failover, automatic load redistribution)
• Cost: Very high (2x+ infrastructure)
• Complexity: Very high (data consistency, conflict resolution, latency)

Best For

Mission-critical systems that can't tolerate any downtime (financial trading, healthcare, emergency services).

graph TD
    U[Users]
    LB[Global Load Balancer]

    subgraph "us-east-1"
    A1[App Servers]
    A2[Database]
    end

    subgraph "us-west-2"
    B1[App Servers]
    B2[Database]
    end

    subgraph "eu-west-1"
    C1[App Servers]
    C2[Database]
    end

    U --> LB
    LB -->|33%| A1
    LB -->|33%| B1
    LB -->|34%| C1

    A2 -.->|Bi-directional replication| B2
    B2 -.->|Bi-directional replication| C2
    C2 -.->|Bi-directional replication| A2

    style A1 fill:#2d5016
    style B1 fill:#2d5016
    style C1 fill:#2d5016

The Problem

Most organizations have DR plans that look great on paper but have never been tested. When real disaster strikes, they find:

• Backups are corrupted
• Runbooks are outdated
• Required credentials are missing
• DR infrastructure can't handle production load
• Critical dependencies are missing

Solution: Regular DR Tests

• Tabletop exercises: Walk through DR plan on paper. Find gaps in documentation.
• Failover tests: Actually failover to DR region. Verify all systems work.
• Full DR simulation: Simulate complete loss of primary region. Measure RTO/RPO.

Frequency: At minimum, test DR annually. Better: quarterly. Best: continuous (automated DR tests).

3-2-1 Rule

• 3 copies: Original + 2 backups
• 2 different media: Disk + tape, or cloud + on-prem
• 1 offsite: Protects against site-wide disasters

Immutable Backups

Protect against ransomware. Once written, backup cannot be modified or deleted for retention period (e.g., 30 days). Even if attacker gains admin access, they can't destroy backups.

Test Restores

Regularly restore from backup to verify integrity. Untested backups are useless. Automate restore tests.

Encrypt Backups

Protect sensitive data. Encrypt at rest and in transit.

In distributed systems, messages can be:

• Duplicated: Retry sends request twice
• Reordered: Network delivers request 2 before request 1
• Lost and retried: Request succeeded but response lost, so client retries

Without idempotency: Retrying can cause duplicates, incorrect state, financial losses.

With idempotency: Safe to retry. System stays consistent.

Client generates unique key for each request. Server stores key after processing. If same key arrives again, return cached response without re-executing.

POST /transfer HTTP/1.1
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Content-Type: application/json

{
  "from": "alice",
  "to": "bob",
  "amount": 100
}

# Server logic
idempotencyKey = request.headers["Idempotency-Key"]
cachedResult = cache.get(idempotencyKey)

if (cachedResult != null) {
  return cachedResult  # Already processed
}

# Process transfer
result = transferMoney(from, to, amount)

# Cache result for 24 hours
cache.set(idempotencyKey, result, ttl=24h)

return result

Key Properties

• Client-generated (UUID, hash of request data)
• Unique per request
• Stable (same key for retry of same request)
• Server stores key + response for TTL (24-48 hours typical)

curl https://api.stripe.com/v1/charges \
  -u sk_test_xxx: \
  -H "Idempotency-Key: 12345" \
  -d amount=2000 \
  -d currency=usd \
  -d source=tok_visa

Design operations to be naturally idempotent by using "set to value" instead of "modify by delta".

Non-Idempotent	Idempotent Alternative
POST /counter/increment	PUT /counter {value: 42}
POST /account/debit {amount: 100}	PUT /transaction/123 {amount: 100} (transaction ID is dedup key)
POST /cart/add {item: "apple"}	PUT /cart {items: ["apple", "banana"]}

Include version number in request. Only apply if current version matches. Prevents applying update twice.

GET /account/123
{
  "balance": 1000,
  "version": 5
}

# Client modifies balance
PUT /account/123
{
  "balance": 900,
  "version": 5  // Expected current version
}

# Server checks
if (account.version == request.version) {
  account.balance = 900
  account.version = 6
  return 200 OK
} else {
  return 409 Conflict {
    "error": "Version mismatch. Account was modified by another request."
  }
}

If retry arrives with version 5, but account is now version 6, server rejects it (already applied).

Use unique constraints to prevent duplicates at database level.

CREATE TABLE transactions (
  id VARCHAR PRIMARY KEY,
  from_account VARCHAR,
  to_account VARCHAR,
  amount DECIMAL,
  created_at TIMESTAMP,
  UNIQUE(from_account, to_account, amount, created_at)
);

# Attempt to insert duplicate transaction
INSERT INTO transactions VALUES ('tx123', 'alice', 'bob', 100, '2024-01-15');
INSERT INTO transactions VALUES ('tx123', 'alice', 'bob', 100, '2024-01-15');

# Second insert fails: duplicate key violation
# Application catches error, knows request already processed

Distributed systems typically provide at-least-once delivery: message may be delivered multiple times, but never lost.

To achieve exactly-once semantics (process message exactly once even if delivered multiple times): make processing idempotent.

# Message queue with at-least-once delivery
while (message = queue.receive()) {
  # Idempotent processing using message ID as dedup key
  if (!processedMessages.contains(message.id)) {
    processMessage(message.data)
    processedMessages.add(message.id)
  }
  queue.ack(message)
}

Even if message delivered twice, it's only processed once.

graph TD
    A[Message Sent]
    B[Message Delivered]
    C{Already
Processed?}
    D[Process Message]
    E[Skip, Already Done]
    F[Ack Message]

    A --> B
    B --> C
    C -->|No| D
    C -->|Yes| E
    D --> F
    E --> F

    style D fill:#2d5016
    style E fill:#4a7ba7

Question: Is the process alive?

What it checks:

• Process is running
• Application can respond to HTTP request

What happens if fails: Restart the process.

Example:

GET /health/liveness
200 OK { "status": "alive" }

Kubernetes Liveness Probe

livenessProbe:
  httpGet:
    path: /health/liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3  # Restart after 3 failures

Question: Is the service ready to accept traffic?

What it checks:

• Process is running (liveness)
• Dependencies are reachable (database, cache)
• Initialization complete (data loaded, connections established)
• Not overloaded (queue depth, CPU usage below threshold)

What happens if fails: Remove from load balancer rotation (stop sending traffic). Don't restart.

Example:

GET /health/readiness
{
  "status": "ready",
  "database": "connected",
  "cache": "connected",
  "queueDepth": 10
}

# If overloaded
{
  "status": "not ready",
  "database": "connected",
  "cache": "connected",
  "queueDepth": 5000  # Too high
}

Kubernetes Readiness Probe

readinessProbe:
  httpGet:
    path: /health/readiness
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 2  # Remove from service after 2 failures

Question: Has the service finished starting up?

Use case: Slow-starting applications (e.g., JVM with large heap, ML models loading).

Why separate from liveness: During startup, app may be slow to respond. Liveness probe would kill it. Startup probe gives more time.

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 10
  failureThreshold: 30  # Allow 300 seconds (5 min) to start

Once startup probe succeeds, liveness and readiness probes take over.

stateDiagram-v2
    [*] --> Starting
    Starting --> Ready: Startup probe passes
    Ready --> NotReady: Readiness probe fails
    NotReady --> Ready: Readiness probe passes
    Ready --> Dead: Liveness probe fails
    NotReady --> Dead: Liveness probe fails
    Dead --> [*]: Restart pod

    Starting: Startup Probe
Checking if initialized
    Ready: Ready Probe
Receiving traffic
    NotReady: Not Ready
No traffic, but alive
    Dead: Liveness Failed
Restart required

Shallow Health Check

Check if process is alive. Fast but not thorough.

GET /health
200 OK { "status": "ok" }

Deep Health Check

Check dependencies, disk space, memory, critical paths.

GET /health/deep
{
  "status": "healthy",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 5,
      "connections": { "active": 10, "max": 100 }
    },
    "cache": {
      "status": "healthy",
      "hit_rate": 0.95
    },
    "disk": {
      "status": "healthy",
      "used_percent": 60
    },
    "memory": {
      "status": "degraded",
      "used_percent": 95
    }
  }
}

When to Use

• Shallow: Liveness probe (called frequently, must be fast)
• Deep: Monitoring dashboards, manual troubleshooting (called infrequently, can be slow)

Use readiness probes to control traffic during deployments.

Canary Deployment

1. Deploy new version (v2) with readiness probe failing
2. No traffic goes to v2 (probe failing)
3. Manually test v2
4. Mark v2 readiness probe as passing
5. Load balancer starts sending small % of traffic to v2
6. Monitor metrics. If good, gradually increase v2 traffic
7. If bad, mark v2 readiness probe as failing. Traffic stops immediately.

Reliability and fault tolerance are about embracing failure. Systems fail constantly. The question is: can your system survive gracefully?

1. Availability Math

• 99.9% = 8.76 hours downtime/year
• 99.99% = 52 minutes downtime/year
• Serial dependencies multiply failure rates: (0.999)^10 = 99.0%
• Use error budgets to balance innovation and stability

2. Failure Types

• Crash failures: easiest to detect and handle
• Omission failures: retries + idempotency
• Timing failures: timeouts + circuit breakers
• Byzantine failures: rare, expensive to handle
• Gray failures: hardest to detect (use percentiles, outlier detection)
• Cascading failures: use bulkheads, load shedding, circuit breakers

3. Redundancy Patterns

• Active-Active: max utilization, instant failover, harder to implement
• Active-Passive: simpler, some wasted capacity, failover delay
• N+2 for production (survives 1 failure during maintenance)
• Hot standby (seconds), warm standby (minutes), cold standby (hours)

4. Core Patterns

• Circuit Breaker: Stop calling failing dependencies, fail fast
• Retry with Backoff + Jitter: Safely retry transient failures
• Bulkhead: Isolate failure domains with separate resource pools
• Timeout: Never wait forever, propagate deadlines

5. Graceful Degradation

• Classify features by criticality (never degrade core, shed nice-to-have)
• Use fallbacks: cached data, default values, reduced quality
• Load shedding: reject some requests to successfully serve others

6. Chaos Engineering

• Test in production (staging != prod)
• Netflix Chaos Monkey: randomly kill instances
• Game days: simulate major outages, test runbooks
• Automate chaos experiments for continuous validation

7. Disaster Recovery

• RPO: how much data loss can you tolerate?
• RTO: how fast must you recover?
• Strategies: backup/restore (cheap, slow) → active-active (expensive, instant)
• Test DR regularly (quarterly minimum)

8. Idempotency

• Make operations safe to retry
• Idempotency keys for non-idempotent operations
• At-least-once delivery + idempotency = exactly-once semantics

9. Health Checking

• Liveness: is process alive? (fail → restart)
• Readiness: can it serve traffic? (fail → stop routing)
• Keep health checks lightweight
• Don't cascade health checks (check local health only)

When asked about reliability in a system design interview, cover:

1. Redundancy: How many instances? Active-active or active-passive? N+2?
2. Failover: How do you detect failures? How do you route traffic away?
3. Circuit Breaker: How do you handle slow/failing dependencies?
4. Retries: Exponential backoff + jitter + idempotency keys
5. Timeouts: Connection, read, write, end-to-end timeouts
6. Graceful Degradation: What features can you drop under load?
7. Health Checks: Liveness vs readiness, what do they check?
8. Disaster Recovery: What's your RPO/RTO? How do you test it?

Key phrase: "Everything fails. We design for resilience by..."

graph TD
    A[Level 5: Chaos Engineering
Proactively test failures]
    B[Level 4: Disaster Recovery
Survive regional outages]
    C[Level 3: Graceful Degradation
Shed load, disable features]
    D[Level 2: Automatic Failover
Circuit breakers, health checks]
    E[Level 1: Redundancy
Multiple instances, replicas]
    F[Level 0: Single Point of Failure
One server, one DB]

    F --> E
    E --> D
    D --> C
    C --> B
    B --> A

    style F fill:#8b0000
    style E fill:#b8860b
    style D fill:#4a7ba7
    style C fill:#2d5016
    style B fill:#2d5016
    style A fill:#2d5016

Start at Level 1: You must have redundancy. No single points of failure.

Mature to Level 2: Automate failover. Don't rely on humans to detect and fix failures.

Advanced to Level 3: Gracefully degrade instead of failing completely.

Production-grade Level 4: Have DR plan and test it.

Elite Level 5: Practice chaos engineering. Break things before they break you.

In Chapter 8: Observability & Monitoring, we'll cover how to see inside your black-box distributed system:

• Metrics, Logs, Traces (the three pillars)
• What to measure (USE, RED, Four Golden Signals)
• Alerting strategies (symptom-based, not cause-based)
• On-call best practices
• Distributed tracing for debugging latency

You can't fix what you can't see. Observability turns invisible distributed systems into transparent, debuggable ones.

Definition: A metastable failure occurs when a system enters a self-sustaining failure state that persists AFTER the initial trigger is removed. Unlike cascading failures (which are progressive), metastable failures are stable in the failed state.

This is the failure mode that separates production-experienced engineers from those who haven't seen true scale. Metastable failures require manual intervention because the system cannot recover on its own.

Normal system: Negative feedback loops stabilize the system (increased load → shed load → reduced load)

Metastable failure: Positive feedback loops amplify the failure (increased load → more retries → even more load)

The failure mode GENERATES the conditions that sustain the failure.

Key Takeaway: Metastable failures are the most insidious failure mode in distributed systems because the system appears "broken" even after you've fixed the root cause. The failure creates its own sustaining load (retry queues, circuit breaker states, thundering herds) that prevents recovery. Always design systems with negative feedback loops (load shedding, rate limiting, dampened recovery) to prevent positive feedback loops that lead to metastable states.