Chapter 19: Load Balancers

1. Why Load Balancing

Core problem: A single server has limited capacity (CPU, memory, network, connections). Load balancers distribute traffic across multiple servers for horizontal scaling, high availability, and single point of failure elimination.

2. Layer 4 vs Layer 7 Load Balancing

Key distinction: L4 operates at transport layer (TCP/UDP), L7 operates at application layer (HTTP/gRPC). Different visibility, capabilities, and performance characteristics.

Direct Server Return (DSR)

L4 optimization where response bypasses load balancer and goes directly from server to client. Reduces LB bandwidth (only ingress traffic passes through LB). Requires server to have LB's VIP on loopback interface.

L4 vs L7 Architecture

3. Load Balancing Algorithms

Goal: Distribute load fairly and efficiently across backend servers. Different algorithms optimize for different goals: simplicity, fairness, statefulness, performance.

Algorithm Comparison

Consistent Hashing Deep Dive

Maps keys and servers to a hash ring (0 to 2^32-1). Each key routes to the next server clockwise on the ring. When a server is added/removed, only ~1/N keys remap (vs 100% with naive hashing).

Virtual nodes: Each physical server gets 100-500 virtual nodes on the ring to ensure even distribution. Without virtual nodes, servers could cluster and cause uneven load.

4. Health Checks

Purpose: Detect unhealthy servers and remove them from the load balancing pool. Prevent routing requests to failed/degraded instances.

Health Check Parameters

Connection Draining

Problem: When server marked unhealthy or removed, in-flight requests should complete gracefully.

Solution: Drain period (30-300s) where no new requests route to server, but existing connections allowed to finish. After drain timeout, forcibly close remaining connections.

Health Check Endpoint Design

// Good: Lightweight, checks critical dependencies
GET /health
Response 200 OK:
{
  "status": "healthy",
  "database": "connected",
  "cache": "connected",
  "uptime": 3600
}

Response 503 Service Unavailable:
{
  "status": "unhealthy",
  "database": "timeout",
  "cache": "connected"
}

// Bad: Too expensive, causes cascading failures
GET /health → triggers full DB query + cache warmup

5. Session Affinity (Sticky Sessions)

Problem: Stateful applications store session data in memory. If user's requests load balance to different servers, session lost.

Solution: Ensure user's requests always route to the same server (sticky sessions), OR externalize session storage (preferred).

Sticky Sessions vs Externalized Sessions

6. SSL Termination

Key decision: Where to decrypt HTTPS traffic? Three models: terminate at LB, pass through to backend, or re-encrypt (bridge mode).

SSL Termination Flow

Certificate Management

7. Global Server Load Balancing (GSLB)

Problem: Users worldwide need to route to nearest datacenter for low latency and high availability across regions.

Solution: DNS-based routing (GeoDNS), Anycast IP, or latency-based routing to direct users to optimal datacenter.

GSLB Architecture

Anycast vs GeoDNS

Failover Strategies

Route 53 Routing Policies

// Latency-based routing
example.com → Route 53
  - US-East: 54.123.45.67 (latency: 10ms from client)
  - EU-West: 18.234.56.78 (latency: 120ms from client)
  → Returns 54.123.45.67

// Geolocation routing
Client from Germany → Returns EU IP
Client from USA → Returns US IP
Client from Brazil → Returns default (US IP)

// Weighted routing (canary / blue-green)
example.com:
  - 90% → Production (v1.0)
  - 10% → Canary (v2.0)

// Failover routing
Primary: US-East (healthy) → Return US IP
Primary: US-East (unhealthy) → Return EU IP

8. Software vs Hardware Load Balancers

Shift: Industry moving from expensive hardware LBs (F5, Citrix) to software LBs (HAProxy, Nginx, Envoy) and cloud-managed (AWS ALB/NLB).

Feature Comparison

Software LB Architecture

9. High Availability of Load Balancers

Problem: Load balancer is a single point of failure. If LB dies, entire system is down.

Solution: Active-passive pairs with VRRP, active-active with DNS round robin, or BGP anycast for multi-LB resilience.

VRRP (Active-Passive) Architecture

VRRP Failover Process

Multi-AZ AWS ALB Architecture

LB HA Comparison

10. Advanced Load Balancing Patterns

Rate Limiting

Protect backend from overload by limiting requests per client. Implemented via token bucket or leaky bucket algorithms.

// Nginx rate limiting
http {
    limit_req_zone $binary_remote_addr zone=api:10m rate=100r/m;

    server {
        location /api/ {
            limit_req zone=api burst=20 nodelay;
            # Allow 100 req/min per IP, burst up to 20
        }
    }
}

Connection Draining

Gracefully remove servers from pool: stop sending new connections, allow existing to complete, timeout after drain period (e.g., 300s).

Canary Deployments

Route small percentage (1-10%) of traffic to new version, monitor errors, gradually increase if healthy.

Blue-Green Deployments

Two identical environments (blue=production, green=new version). Deploy to green, test, then switch LB to route 100% to green.

A/B Testing via LB

Route users to different versions based on criteria (header, cookie, user ID) to compare behavior/performance.

// Envoy A/B routing
routes:
  - match:
      prefix: "/api"
      headers:
        - name: "X-Experiment"
          exact_match: "variant-b"
    route:
      cluster: service_variant_b
  - match:
      prefix: "/api"
    route:
      weighted_clusters:
        clusters:
          - name: service_variant_a
            weight: 50
          - name: service_variant_b
            weight: 50

Circuit Breaking at LB

LB detects backend failure patterns (high error rate, timeouts) and temporarily stops routing to unhealthy backend, preventing cascading failures.

Retry and Timeout Policies

Real-World Configurations & Benchmarks

Performance Benchmarks

AWS Limits & Characteristics

Failure Timeline Example

Scenario: Backend server crashes behind a load balancer.