Chapter 43: Global Distribution & Geo-Replication

1. Why Go Global

1.1 Latency: The Physics Problem

Speed of light in fiber optic cable: ~200,000 km/s (67% of speed of light in vacuum). Round-trip latency is determined by physical distance:

Impact: A user in Tokyo accessing a US-only service experiences 100ms base latency before any processing. For interactive applications (target: <100ms), this is unacceptable. Multi-region deployment places compute and data closer to users.

1.2 Availability: Survive Region Failures

AWS regions have failed (us-east-1 outages: 2017, 2020, 2021, 2022). A single-region deployment has a single point of failure. Multi-region architecture provides disaster recovery:

• Active-Passive: Failover to standby region (RPO: minutes to hours, RTO: minutes)
• Active-Active: Both regions serve traffic simultaneously (RPO: seconds, RTO: seconds)

1.3 Compliance: Data Residency Requirements

Regulations mandate data storage location:

• GDPR (EU): Personal data of EU residents must remain in EU or approved countries
• China Cybersecurity Law: Critical data must be stored in mainland China
• Russian Data Localization: Personal data of Russian citizens must be stored in Russia
• Financial regulations: Many require in-country data storage (India, Indonesia, Vietnam)

Multi-region deployment with data residency controls allows global service while meeting local laws.

2. Multi-Region Architecture Patterns

2.1 Active-Passive (Disaster Recovery)

Characteristics:

• PRO: Simplest to implement, lowest cost (only one active region)
• PRO: No conflict resolution needed (single write source)
• CON: Higher latency for distant users (all traffic to one region)
• CON: Failover requires DNS change or manual intervention (minutes RTO)

Use case: Cost-sensitive applications with acceptable downtime during regional outages.

2.2 Active-Active (Multi-Master)

Characteristics:

• PRO: Lowest latency (users routed to nearest region)
• PRO: Highest availability (automatic failover, no RTO)
• CON: Complex conflict resolution (multiple write sources)
• CON: Higher cost (full stack in every region + cross-region bandwidth)

Use case: Global consumer applications where latency and availability are critical.

2.3 Follow-the-Sun

Characteristics:

• PRO: Single active region avoids conflicts
• PRO: Lower cost than active-active
• NEUTRAL: Works for workloads with natural daily cycles
• CON: Complex orchestration (scheduled failover)

Use case: Internal tools with business-hours traffic (support dashboards, batch processing).

Pattern Comparison

3. Data Replication Strategies

3.1 Synchronous Replication

Characteristics:

• PRO: Zero data loss on primary failure (replica always up-to-date)
• PRO: Strong consistency (all replicas see same data)
• CON: High write latency (must wait for cross-region network)
• CON: Reduced availability (write fails if replica unreachable)

Example: Google Cloud Spanner (uses Paxos for synchronous replication across zones).

3.2 Asynchronous Replication

Characteristics:

• PRO: Low write latency (no cross-region wait)
• PRO: High availability (writes succeed even if replica down)
• CON: Eventual consistency (replicas lag behind primary)
• CON: Data loss on primary failure (uncommitted writes lost)

Example: Most databases (MySQL replication, PostgreSQL streaming replication, MongoDB replica sets).

3.3 Semi-Synchronous Replication

Hybrid approach: primary waits for at least one replica ACK before committing. Balances durability and latency.

• PRO: Reduced data loss (at least one replica has data)
• NEUTRAL: Write latency = nearest replica RTT (not all replicas)
• CON: Still eventual consistency across all replicas

Example: MySQL semi-sync replication, PostgreSQL with synchronous_standby_names=1.

Latency vs. Consistency Trade-off

Rule of thumb: If your application requires reads to always see the latest write (read-your-writes consistency), you must either (a) use synchronous replication, or (b) route reads to the primary until replication completes.

4. Conflict Resolution

In active-active deployments, concurrent writes to the same key in different regions create conflicts. Strategies to resolve:

4.1 Last-Writer-Wins (LWW)

Each write tagged with timestamp. During conflict, highest timestamp wins.

Region 1: WRITE(key=X, val=A, ts=1000)
Region 2: WRITE(key=X, val=B, ts=1001)
Result: X=B (ts=1001 > ts=1000)

• PRO: Simple, deterministic
• CON: Data loss (value A discarded)
• CON: Relies on synchronized clocks (clock skew causes issues)

Use case: DynamoDB Global Tables, Cassandra (with time-based conflict resolution).

4.2 Vector Clocks

Each replica maintains a counter for every other replica. Detects causality:

Node A: writes X=1, vector=[A:1, B:0, C:0]
Node B: writes X=2, vector=[A:1, B:1, C:0]
Node C: writes X=3, vector=[A:1, B:0, C:1]

B's write causally depends on A (A:1 ≤ B:1).
C's write concurrent with B (neither vector dominates).

• PRO: Accurate causality tracking (no reliance on clocks)
• CON: Vector grows with number of replicas (metadata overhead)
• CON: Application must handle concurrent conflicts

Use case: Riak, early Dynamo (deprecated in favor of LWW for simplicity).

4.3 CRDTs (Conflict-Free Replicated Data Types)

Data structures designed to merge conflicts automatically. Types:

• G-Counter: Grow-only counter (each replica increments its own counter, sum for total)
• PN-Counter: Increment/decrement counter (two G-Counters: P for increments, N for decrements)
• LWW-Element-Set: Set with add/remove (each element tagged with timestamp)
• OR-Set: Observed-Remove Set (add wins over remove if concurrent)

Example: Redis CRDT (Redis Enterprise), Riak Data Types.

• PRO: Automatic conflict resolution (no application logic needed)
• CON: Limited data types (counters, sets, registers—not arbitrary SQL)
• CON: Cannot express all business logic (e.g., "decrement only if positive")

4.4 Application-Level Merge

Database stores all conflicting values. Application decides how to merge:

Conflict: Shopping cart updated in two regions
Region 1: Add [ItemA]
Region 2: Add [ItemB]
Application merge: Union → [ItemA, ItemB] (both added)

• PRO: Full control (business logic for conflict resolution)
• CON: Complex application code

Use case: Collaborative editing (Google Docs uses Operational Transformation, similar concept).

4.5 Conflict-Free by Design

Architect system so conflicts cannot occur:

• Partition data by region: Each region owns a subset (US users in us-east, EU users in eu-west)
• Immutable data: Only append, never update (event sourcing)
• Single writer per key: Use consistent hashing to route writes for a key to one region

Example: Netflix (partitions data by customer region, cross-region replication for reads only).

5. Traffic Routing

How to direct users to the optimal region:

5.1 GeoDNS (DNS-Based Routing)

DNS server returns different IP addresses based on client's geographic location:

User in US queries api.example.com → 34.200.1.1 (us-east-1)
User in EU queries api.example.com → 52.48.2.2 (eu-west-1)

• PRO: Simple, widely supported
• CON: DNS caching delays failover (TTL typically 60s-300s)
• CON: Coarse-grained (by country, not city)

Provider: AWS Route 53 (geolocation routing), Cloudflare Load Balancing.

5.2 Anycast

Same IP address announced from multiple locations. BGP routes traffic to nearest location:

• PRO: Lowest latency (network-level routing to nearest POP)
• PRO: DDoS mitigation (traffic absorbed at edge)
• CON: Stateless only (TCP connections may break on BGP changes)
• CON: Requires ISP cooperation (must announce IPs via BGP)

Use case: CDNs (Cloudflare, Fastly), DNS servers (1.1.1.1, 8.8.8.8).

5.3 Latency-Based Routing

DNS returns IP of region with lowest latency to user (measured via probes):

• PRO: More accurate than geolocation (accounts for network topology)
• CON: Still DNS-based (TTL caching delays failover)

Provider: AWS Route 53 latency-based routing.

5.4 Geoproximity Routing

Like geolocation but with "bias" to shift traffic boundaries:

Region 1 (US): bias=+20 (expand coverage)
Region 2 (EU): bias=-10 (shrink coverage)
→ Some EU users routed to US (load balancing)

• PRO: Fine-grained traffic shaping
• CON: Complex to tune

Provider: AWS Route 53 geoproximity.

5.5 Failover Routing

Primary region serves traffic. If health check fails, DNS switches to secondary:

Primary: us-east-1 (healthy) → serve traffic
Primary: us-east-1 (unhealthy) → failover to eu-west-1

• PRO: Automatic disaster recovery
• CON: Failover delayed by DNS TTL (60s-300s)

6. Data Residency & Compliance

6.1 GDPR (General Data Protection Regulation)

Requirement: Personal data of EU residents must be stored in EU or countries with "adequate" data protection (EU Commission approved list: UK, Switzerland, Japan, Canada—not US after Privacy Shield invalidation).

Implementation:

• Store EU user data in eu-west-1, eu-central-1, or other EU regions
• Use AWS S3 block public access + bucket policies to prevent cross-region replication
• Encrypt data at rest (GDPR Article 32: "appropriate security measures")

// Enforce data residency in code
if (user.region === 'EU') {
  databaseConnection = euWestConnection; // eu-west-1
} else if (user.region === 'US') {
  databaseConnection = usEastConnection; // us-east-1
}

6.2 China Cybersecurity Law

Requirement: Critical information infrastructure operators must store personal information and important data within mainland China. Cross-border transfer requires security assessment.

Implementation:

• Use AWS China regions (operated by Sinnet, separate from global AWS)
• Deploy full stack in China (cannot rely on cross-border APIs)
• Data isolation (no replication to non-China regions)

6.3 Russia Data Localization Law

Requirement: Personal data of Russian citizens must be stored on servers in Russia.

Challenge: No AWS regions in Russia (closest: eu-central-1 Frankfurt). Must use local providers or on-prem.

6.4 Financial Services Regulations

Many countries require financial data to remain in-country:

• India: RBI mandates payment data stored in India
• Indonesia: Bank Indonesia requires financial data in Indonesia
• Vietnam: Circular 22 requires data stored in Vietnam

Implementation Patterns

Pattern 1: Region-Pinned Data

users table:
  id, email, name, region_lock (enum: EU, US, ASIA)

SELECT * FROM users WHERE id=123 AND region_lock='EU';
// Application enforces: EU data only queried from EU database

Pattern 2: Separate Databases per Region

// No cross-region replication
eu_db = postgres://eu-west-1/users
us_db = postgres://us-east-1/users
asia_db = postgres://ap-south-1/users

// Shard by user region
db = get_db_for_region(user.region)

Pattern 3: Tokenization for Cross-Border

// Store sensitive data in-region, reference token globally
EU DB: { user_id: 123, name: "John", ssn: "encrypted" }
Global DB: { user_id: 123, name_token: "ref_eu_123" }

// Global analytics use tokens, not raw data

7. Global Database Options

7.1 Google Cloud Spanner

Consistency: Strong (linearizable, external consistency via TrueTime)

Replication: Synchronous (Paxos across zones/regions)

Write Latency: 50-200ms (depends on replication distance)

Use case: Global applications requiring strong consistency (financial systems, inventory).

// Spanner multi-region config
CREATE DATABASE mydb
  WITH REGIONS = ['us-central1', 'us-east1', 'europe-west1']
  READ_WRITE_TRANSACTION_MODE = STRONG;

• PRO: No conflict resolution (strong consistency)
• PRO: Horizontal scalability (1000s nodes)
• CON: High write latency (cross-region sync)
• CON: Expensive (starts at $300/month per node)

7.2 CockroachDB

Consistency: Serializable (Raft consensus)

Replication: Synchronous (configurable replicas per zone)

Write Latency: 50-150ms (majority quorum)

Use case: Open-source alternative to Spanner (can run on any cloud or on-prem).

// CockroachDB multi-region setup
ALTER DATABASE mydb SET PRIMARY REGION 'us-east1';
ALTER DATABASE mydb ADD REGION 'us-west1';
ALTER DATABASE mydb ADD REGION 'eu-west1';

ALTER TABLE users SET LOCALITY REGIONAL BY ROW;
// Each row pinned to region (GDPR compliance)

• PRO: Strong consistency (serializable isolation)
• PRO: Open-source (self-hosted option)
• CON: Write latency (synchronous replication)

7.3 AWS DynamoDB Global Tables

Consistency: Eventual (last-writer-wins conflict resolution)

Replication: Asynchronous (typically <1 second lag)

Write Latency: 5-10ms (local write, async replication)

Use case: High-throughput applications with eventual consistency tolerance.

// DynamoDB Global Table setup
aws dynamodb create-global-table \
  --global-table-name Users \
  --replication-group RegionName=us-east-1 RegionName=eu-west-1

• PRO: Low write latency (async replication)
• PRO: Fully managed (auto-scaling, no ops)
• CON: Eventual consistency (conflicts resolved via LWW)
• CON: Limited query flexibility (NoSQL)

7.4 Azure Cosmos DB

Consistency: 5 levels (strong, bounded staleness, session, consistent prefix, eventual)

Replication: Configurable (multi-master or single-master)

Write Latency: Varies by consistency (strong: 50-200ms, eventual: 5-10ms)

Use case: Applications needing tunable consistency (adjust per query).

// Cosmos DB consistency levels
Strong: Linearizability (like Spanner)
Bounded Staleness: Reads lag by max K versions or T seconds
Session: Read-your-writes (default for most apps)
Consistent Prefix: Reads never see out-of-order writes
Eventual: Lowest latency, highest availability

• PRO: Flexibility (5 consistency choices)
• PRO: Multi-model (SQL, MongoDB, Cassandra APIs)
• CON: Complexity (choosing consistency is hard)
• CON: Cost (expensive for high throughput)

Comparison Table

8. Caching Globally

8.1 Regional Cache Clusters

Deploy cache (Redis, Memcached) in every region to minimize latency:

Challenge: Cache invalidation across regions. When data changes, must invalidate cache in all regions.

8.2 Cache Invalidation Strategies

Strategy 1: Pub/Sub Invalidation

// On write to database (us-east-1)
db.update(user_id=123, name="Alice")
pubsub.publish("cache_invalidate", {key: "user:123"})

// All regions subscribe
redis_us.subscribe("cache_invalidate", (msg) => redis_us.del(msg.key))
redis_eu.subscribe("cache_invalidate", (msg) => redis_eu.del(msg.key))

• PRO: Fast invalidation (100-500ms cross-region)
• CON: Requires pub/sub infrastructure (SNS, Redis Pub/Sub)

Strategy 2: Short TTL

// Cache with 30-60 second TTL
redis.set("user:123", userData, EX=60) // expires in 60s

// Stale data tolerated for 60s max

• PRO: Simple (no invalidation logic)
• CON: Higher cache miss rate (frequent expiration)

Strategy 3: Versioned Keys

// Include version in cache key
cache_key = f"user:{user_id}:v{db_version}"
redis.set(cache_key, userData)

// On update, increment version
db_version += 1
// Old cache keys naturally become stale (never accessed)

• PRO: No explicit invalidation
• CON: Cache bloat (old versions consume memory until TTL expires)

8.3 Warm-Up Strategies

When deploying new region or recovering from failure, cache is empty (cold start):

• Pre-warm: Before routing traffic, populate cache with hot keys (top 1000 users, products)
• Gradual ramp: Route 1% → 5% → 10% → 100% traffic over hours (allows cache to build organically)
• Cache-aside with fallback: On cache miss, fetch from primary region's cache (not database) to reduce DB load

9. Failure Modes

9.1 Region Failure

Scenario: us-east-1 becomes unavailable (power outage, network partition).

Active-Passive:

• Manual or automated failover to standby region (eu-west-1)
• RTO: 5-30 minutes (depends on DNS TTL, health checks, runbook)
• RPO: 1-5 minutes (asynchronous replication lag)

Active-Active:

• Traffic automatically reroutes to healthy regions
• RTO: Seconds (health check interval + DNS TTL or anycast BGP convergence)
• RPO: Seconds (in-flight writes to failed region lost)

Mitigation: Use multi-region architecture, regularly test failover (Chaos Engineering).

9.2 Split-Brain

Scenario: Network partition causes two regions to operate independently, both accepting writes.

Consequence: Conflicting writes. When partition heals, conflict resolution required.

Mitigation:

• Quorum-based writes: Require majority of replicas (prevents split-brain)
• Fencing: Disable failed region (STONITH: "shoot the other node in the head")
• Conflict-free design: Partition data so each region owns a subset (no overlap)

9.3 Replication Lag Amplification

Scenario: Primary region overloaded. Replication lag grows from 1s → 10s → 60s.

Normal: Primary writes 1000 TPS, replica lags 1s
Spike: Primary writes 5000 TPS, replica lags 30s (falling behind)
Failure: Replica never catches up (lag grows unbounded)

Consequence: Stale reads, data loss on failover (uncommitted writes).

Mitigation:

• Monitor replication lag (alert if >5s)
• Scale replica compute (increase IOPS, CPU)
• Reduce write rate (backpressure, rate limiting)
• Parallel replication (multiple threads applying binlog)

9.4 Thundering Herd on Failover

Scenario: Region 1 fails. All traffic instantly reroutes to Region 2.

Before: Region 1 (10,000 RPS), Region 2 (10,000 RPS)
After: Region 1 (offline), Region 2 (20,000 RPS) → overload → cascading failure

Mitigation:

• Over-provision: Each region can handle 2x normal load (50% headroom)
• Gradual failover: Ramp traffic over minutes (not instant)
• Circuit breaker: Shed load if Region 2 overloaded (fail fast)
• Autoscaling: Trigger aggressive scale-up on failover

10. Cost Considerations

10.1 Cross-Region Data Transfer

Cloud providers charge for data egress between regions:

Example calculation:

Scenario: 100 GB/day replication (us-east-1 → eu-west-1)
AWS cost: 100 GB × $0.02 = $2/day = $60/month per direction
Bidirectional: $120/month

Scale: 1 TB/day = $600/month per direction = $1200/month total

Mitigation:

• Replicate only essential data (not logs, temp files)
• Compress data before transfer (reduce bandwidth by 50-70%)
• Use AWS PrivateLink or VPC peering (slightly cheaper than public internet)
• Batch replication (hourly instead of real-time)

10.2 Replication Storage Cost

Each region stores a full copy of data:

Single region: 10 TB data = $230/month (S3 Standard)
Three regions: 30 TB data = $690/month

Database: 10 TB RDS storage = $1,150/month (gp3)
Three regions: 30 TB = $3,450/month

Mitigation:

• Use tiered storage (hot data in all regions, cold data in one region + S3 cross-region replication)
• Selective replication (user data replicated, logs stored locally)
• Compress historical data

10.3 Compute Duplication

Active-active requires full compute stack in every region:

Single region: 10 × c5.2xlarge = $1,530/month
Three regions: 30 instances = $4,590/month

Plus: Load balancers, NAT gateways, etc. (×3 regions)

Mitigation:

• Active-passive (only one active region, standby is minimal)
• Autoscaling (scale down during off-peak)
• Spot instances for non-critical workloads

10.4 Hidden Costs

• NAT Gateway: $0.045/GB processed (×3 regions = 3× cost)
• VPC Peering: $0.01/GB transfer (if using cross-region peering)
• CloudWatch Logs: $0.50/GB ingestion (×3 regions = 3× cost)
• DNS queries: Route 53 charges per query (scales with traffic)

Cost Optimization Strategies