Chapter 5: Latency, Throughput, and Bandwidth

Numbers Every Engineer Should Know

Introduction

System design interviews and real-world architecture decisions hinge on understanding performance characteristics. You need to know whether a given design can handle 1,000 users or 1 billion, whether adding a cache saves 10ms or 100ms, and when vertical scaling stops working.

This chapter provides:

  • Latency numbers: How long operations take (L1 cache vs disk vs network)
  • Throughput numbers: How much you can process per second (IOPS, QPS, bandwidth)
  • Estimation frameworks: Converting user requirements into concrete numbers
  • Laws and principles: Little's Law, Amdahl's Law, tail latency amplification
Interview Tip: Interviewers expect you to make reasonable assumptions and do rough calculations. Showing the process matters more than getting exact numbers. Use powers of 2 and round aggressively.

Jeff Dean's Latency Numbers (2024/2025 Updated)

Jeff Dean's famous "Numbers Everyone Should Know" talk quantified operation costs. Here are updated values reflecting modern hardware:

Operation Latency Scaled to Human Time
L1 cache reference 1 ns 1 second
Branch mispredict 3 ns 3 seconds
L2 cache reference 4 ns 4 seconds
L3 cache reference 10 ns 10 seconds
Mutex lock/unlock 17 ns 17 seconds
Main memory reference 100 ns 1.5 minutes
Compress 1KB with Snappy 3 ฮผs (3,000 ns) 50 minutes
Send 1KB over 1 Gbps network 10 ฮผs 2.7 hours
SSD random read (4KB) 16 ฮผs 4.4 hours
Read 1MB sequentially from SSD 49 ฮผs 13.6 hours
Network round trip in same datacenter 500 ฮผs 5.8 days
Read 1MB sequentially from HDD 825 ฮผs 9.5 days
HDD disk seek 2-10 ms 23-115 days
Send packet CA โ†’ Netherlands โ†’ CA 150 ms 4.8 years
graph LR A[L1 Cache: 1ns] -->|4x| B[L2 Cache: 4ns] B -->|2.5x| C[L3 Cache: 10ns] C -->|10x| D[RAM: 100ns] D -->|30x| E[Compress 1KB: 3ฮผs] E -->|5x| F[SSD Read: 16ฮผs] F -->|50x| G[Network: 500ฮผs] G -->|4x| H[HDD Read: 2ms] H -->|75x| I[Cross-Continent: 150ms] style A fill:#2e7d32 style B fill:#388e3c style C fill:#43a047 style D fill:#66bb6a style E fill:#fbc02d style F fill:#f57c00 style G fill:#e64a19 style H fill:#c62828 style I fill:#6a1b9a
Key Insight: Memory is 100x slower than L1 cache. SSD is 160x slower than memory. Network is 5000x slower than memory. HDD seek is 20,000x slower than memory. Cross-continent network is 1,500,000x slower than memory.

Visualizing the Scale

If L1 cache access = 1 second, then:

L1 cache:        โ– 1 second
L2 cache:        โ– 4 seconds
L3 cache:        โ–Š 10 seconds
RAM:             โ–ˆโ–ˆโ–ˆโ–ˆโ–Œ 1.5 minutes
SSD:             โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 4.4 hours
Network (DC):    โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 5.8 days
HDD:             โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 9.5 days
Cross-continent: [Visualization would span 4.8 years]

Practical Implications

flowchart TD A[Request Arrives] --> B{In L1/L2 Cache?} B -->|Yes: 1-4ns| C[Return Immediately] B -->|No| D{In RAM?} D -->|Yes: 100ns| E[Load to Cache] D -->|No| F{On SSD?} F -->|Yes: 16ฮผs| G[Load to RAM] F -->|No| H{On Remote Server?} H -->|Yes: 500ฮผs-150ms| I[Network Fetch] H -->|No| J{On HDD?} J -->|Yes: 2-10ms| K[Disk Seek + Read] style C fill:#2e7d32 style E fill:#66bb6a style G fill:#f57c00 style I fill:#e64a19 style K fill:#c62828
Interview Tip: When asked "should we cache this?", calculate latency saved ร— frequency. If you save 150ms on a cross-continent call that happens 10,000 times/sec, that's 1,500 seconds = 25 minutes of latency saved per second. Caching is a no-brainer.

Common Mistakes

  • Confusing bandwidth and latency: A fast network (10 Gbps) still has RTT latency (speed of light)
  • Ignoring seek time: HDDs can do 100MB/s sequential but only 100-200 random IOPS
  • Forgetting compression costs: Compressing 1KB takes 3ฮผs but can save 10ฮผs of network time
  • Assuming network = LAN: Same-datacenter is 500ฮผs, cross-continent is 150ms (300x worse)

Throughput Numbers

Latency tells you how long one operation takes. Throughput tells you how many operations you can do per second.

Network Throughput

Connection Type Bandwidth Real-World Throughput Use Case
Home Broadband 100 Mbps ~12 MB/s Streaming, remote work
Gigabit Ethernet (1G NIC) 1 Gbps ~125 MB/s Standard server NIC
10G NIC 10 Gbps ~1.25 GB/s High-traffic web servers
25G NIC 25 Gbps ~3.1 GB/s Cloud instances (AWS c5n, m5n)
40G NIC 40 Gbps ~5 GB/s Storage backends, databases
100G NIC 100 Gbps ~12.5 GB/s Spine switches, high-freq trading
Conversion Formula:
Throughput (MB/s) = Bandwidth (Gbps) ร— 125
Why 125? 1 Gbps = 1,000 Mbps รท 8 bits/byte = 125 MB/s
Real-world: Subtract ~5-10% for TCP/IP overhead
graph LR A[1 Gbps NIC] -->|125 MB/s| B[10 Photos/sec
12 MB each] A -->|125 MB/s| C[1 HD Video Stream
5 Mbps] A -->|125 MB/s| D[~125,000 API Responses/sec
1KB each] E[10 Gbps NIC] -->|1.25 GB/s| F[100 HD Video Streams] E -->|1.25 GB/s| G[~1.25M API Responses/sec] style A fill:#1976d2 style E fill:#0d47a1

Disk Throughput

Storage Type Sequential Read Random Read IOPS Latency
HDD (7200 RPM) 100-200 MB/s 75-100 IOPS 5-10 ms
SATA SSD 500-550 MB/s ~90K IOPS 100-500 ฮผs
NVMe SSD 3-7 GB/s 500K-1M IOPS 10-100 ฮผs
AWS EBS gp3 125-1000 MB/s 3K-16K IOPS (configurable) ~1 ms (variable)
AWS EBS io2 1000 MB/s Up to 64K IOPS Sub-ms
In-Memory (RAM) ~50 GB/s (DDR4) Millions 100 ns
graph TD A[Storage Hierarchy] --> B[RAM: 50 GB/s] A --> C[NVMe: 7 GB/s] A --> D[SATA SSD: 550 MB/s] A --> E[HDD: 150 MB/s] B --> B1[Millions IOPS
100ns latency] C --> C1[1M IOPS
10-100ฮผs latency] D --> D1[90K IOPS
100-500ฮผs latency] E --> E1[100 IOPS
5-10ms latency] style B fill:#2e7d32 style C fill:#66bb6a style D fill:#f57c00 style E fill:#c62828
Interview Tip: If the workload is random reads (e.g., key-value store), focus on IOPS, not sequential throughput. An HDD can push 200 MB/s sequentially but only ~100 random reads/sec. SSD does 500K random reads/sec.

Database Throughput

Database Throughput Notes
PostgreSQL ~50K TPS Single instance, read-heavy
MySQL ~40K TPS InnoDB, write-heavy
Redis (single-threaded) 100K-200K ops/s In-memory, simple ops
Redis (clustered) 1M+ ops/s Sharded across nodes
Cassandra ~100K writes/s per node Write-optimized, scales linearly
MongoDB ~50K writes/s Document store, single node
Elasticsearch ~20K docs/s per node Indexing throughput
DynamoDB Unlimited (provisioned) Pay for throughput units
Rule of Thumb: A single database instance handles ~10K-100K QPS depending on query complexity. Beyond that, you need read replicas, sharding, or a distributed database.

Single-Threaded Compute

Operation Throughput (single core)
Simple arithmetic (add) ~1B ops/sec (1 GHz CPU, 1 cycle/op)
SHA-256 hash ~500K hashes/sec
Bcrypt hash (cost 10) ~100 hashes/sec
JSON parse (1KB object) ~100K ops/sec
Regex match (complex) ~50K matches/sec
gzip compress (1KB) ~300K ops/sec
Interview Tip: When asked "can one server handle this load?", estimate the per-request CPU cost. If a request takes 10ms of CPU and you have 4 cores, you can handle 400 requests/sec. Add 20% safety margin โ†’ 320 req/sec.

Storage Math

Capacity planning requires translating user behavior into bytes. Here's the math you need.

Unit Conversions

Unit Bytes Power of 2 Example
1 Byte 1 2^0 Single ASCII character
1 KB 1,024 2^10 Small text file
1 MB 1,048,576 2^20 Photo (low res)
1 GB 1,073,741,824 2^30 HD movie (compressed)
1 TB 1,099,511,627,776 2^40 200 HD movies
1 PB 1,125,899,906,842,624 2^50 Facebook's daily ingestion
Interview Shortcut: Use powers of 10 for rough estimates:
1 KB โ‰ˆ 10^3 bytes
1 MB โ‰ˆ 10^6 bytes
1 GB โ‰ˆ 10^9 bytes
1 TB โ‰ˆ 10^12 bytes
Error is ~5%, acceptable for interviews.

Common Entity Sizes

Entity Size Notes
Tweet (280 chars) ~560 bytes UTF-8 encoding (2 bytes/char avg)
Facebook post (text only) ~2 KB Includes metadata
User profile (minimal) ~1 KB Name, email, password hash
User profile (with bio, settings) ~10 KB Extended fields
Photo (smartphone, compressed) 2-5 MB JPEG, 12MP camera
Photo thumbnail ~20 KB 200x200 pixels
Video (HD, 1 minute) ~150 MB 1080p, H.264
Video (4K, 1 minute) ~400 MB 3840x2160, H.265
Audio (MP3, 1 minute) ~1 MB 128 kbps bitrate
graph TD A[Data Types by Size] --> B[Text: Bytes-KB] A --> C[Images: KB-MB] A --> D[Videos: MB-GB] B --> B1[Tweet: 560 bytes] B --> B2[Post: 2 KB] B --> B3[Profile: 10 KB] C --> C1[Thumbnail: 20 KB] C --> C2[Photo: 3 MB] C --> C3[Icon: 5 KB] D --> D1[1min HD: 150 MB] D --> D2[1min 4K: 400 MB] D --> D3[Movie: 2 GB] style B fill:#4caf50 style C fill:#ff9800 style D fill:#f44336

Worked Example: Instagram Storage

Assumptions:

  • 500M daily active users (DAU)
  • Each user uploads 2 photos/day
  • Each photo is 3 MB (original)
  • Store 3 compressed versions: thumbnail (20 KB), medium (200 KB), original (3 MB)
  • Retention: 10 years

Calculation:

Photos per day = 500M users ร— 2 photos = 1 billion photos/day

Storage per photo = 20 KB + 200 KB + 3 MB
                  โ‰ˆ 3.22 MB

Daily storage = 1B photos ร— 3.22 MB
              = 3.22 PB/day

Yearly storage = 3.22 PB ร— 365
               โ‰ˆ 1,175 PB
               โ‰ˆ 1.15 EB/year

10-year storage = 11.5 EB
Interview Tip: After calculating raw storage, mention:
1. Replication factor: 3x for durability โ†’ 34.5 EB
2. Compression: Object storage like S3 may compress further (10-20% savings)
3. Cold storage: Move old photos to Glacier (90% cost savings)
4. Deduplication: Same photo uploaded by multiple users

Storage Cost Estimation

Service Cost per GB/month Use Case
AWS S3 Standard $0.023 Hot data, frequent access
AWS S3 Infrequent Access $0.0125 Accessed monthly
AWS S3 Glacier $0.004 Archival, rare access
AWS EBS (SSD) $0.10 Block storage for EC2
GCP Cloud Storage $0.020 Object storage

Instagram 10-year cost (11.5 EB):

Raw storage: 11.5 EB = 11,500,000 TB = 11,500,000,000 GB

S3 Standard cost = 11.5B GB ร— $0.023/GB
                 = $264M/month
                 = $3.17B/year

With tiering:
- Year 1 photos (hot): 1.15 EB ร— $0.023 = $26.4M/month
- Year 2-3 photos (warm): 2.3 EB ร— $0.0125 = $28.75M/month
- Year 4-10 photos (cold): 8.05 EB ร— $0.004 = $32.2M/month

Total: ~$87M/month = $1.05B/year (67% savings)

Quick Estimation Table

Scenario Calculation Result
1M users, 1KB profile each 10^6 ร— 10^3 bytes 1 GB
1B tweets/day, 500 bytes each 10^9 ร— 500 500 GB/day
100M photos/day, 3MB each 10^8 ร— 3 MB 300 TB/day
1M videos/day, 150MB each 10^6 ร— 150 MB 150 TB/day

Back-of-Envelope Estimation Framework

Interviewers want to see your process, not perfect numbers. Follow this framework:

flowchart TD A[Start: Problem Statement] --> B[Clarify Scope] B --> C[Estimate Users] C --> D[Calculate Traffic] D --> E[Estimate Storage] E --> F[Calculate Bandwidth] F --> G[Determine Server Count] G --> H[Validate Assumptions] B --> B1[DAU? MAU? Global?] C --> C1[Peak vs average load] D --> D1[Read/write ratio] E --> E1[Retention policy] F --> F1[Upload vs download] G --> G1[Single server capacity] style A fill:#1976d2 style H fill:#2e7d32

Step 1: Clarify Scope

Ask questions before diving in:

  • How many users? (DAU, MAU, registered vs active)
  • Read-heavy or write-heavy?
  • What operations? (Post, comment, search, upload video)
  • Latency requirements? (Real-time chat vs batch analytics)
  • Availability requirements? (99.9% vs 99.999%)

Step 2: Estimate Traffic (QPS)

DAU โ†’ QPS Formula:

Average QPS = (DAU ร— actions per user per day) / 86,400 seconds

Peak QPS = Average QPS ร— 2-3

Why 86,400? Seconds in a day (24h ร— 60m ร— 60s)

Example: Twitter

Assumptions:
- 200M DAU
- Each user reads 100 tweets/day (reads)
- Each user posts 2 tweets/day (writes)

Read QPS = (200M ร— 100) / 86,400
         = 20B / 86,400
         โ‰ˆ 231,000 reads/sec

Write QPS = (200M ร— 2) / 86,400
          = 400M / 86,400
          โ‰ˆ 4,630 writes/sec

Peak Read QPS = 231K ร— 2 = 462K
Peak Write QPS = 4.6K ร— 2 = 9.2K
Interview Tip: Use 100K seconds/day instead of 86,400 for mental math. It's 15% off but much easier: 200M ร— 100 / 100K = 200K reads/sec. Close enough!

Step 3: Estimate Storage

Storage Formula:

Daily Storage = writes per day ร— size per write

Total Storage = Daily Storage ร— retention days ร— replication factor

Example: Twitter (continued)

Tweets per day = 200M users ร— 2 tweets = 400M tweets/day

Size per tweet = 280 chars ร— 2 bytes + metadata
               โ‰ˆ 560 bytes + 200 bytes
               = ~1 KB (round up)

Daily storage = 400M ร— 1 KB = 400 GB/day

5-year storage = 400 GB ร— 365 ร— 5
               = 730 TB

With 3x replication = 2.2 PB

Step 4: Estimate Bandwidth

Bandwidth Formula:

Bandwidth (Mbps) = QPS ร— response size (bytes) ร— 8 bits/byte / 10^6

Or simplified: Bandwidth (Gbps) โ‰ˆ QPS ร— KB per response / 125

Example: Twitter (continued)

Peak read QPS = 462K
Response size = 20 tweets ร— 1 KB = 20 KB (JSON with metadata)

Bandwidth = 462K ร— 20 KB / 125
          โ‰ˆ 74 Gbps

That's ~3 servers with 25 Gbps NICs (with headroom).

Step 5: Estimate Server Count

Server Count Formula:

Servers = (Peak QPS / single server capacity) ร— safety factor

Safety factor = 1.5-2.0 (for failures, rolling deploys, traffic spikes)

Example: Twitter (continued)

Assume one server handles 10K QPS (after caching, optimized DB queries).

Servers for reads = 462K / 10K ร— 1.5
                  = 46 ร— 1.5
                  โ‰ˆ 70 read servers

Servers for writes = 9.2K / 10K ร— 1.5
                   = 1 ร— 1.5
                   โ‰ˆ 2 write servers

Database:
- Single PostgreSQL handles ~50K TPS
- Read load: 462K QPS โ†’ need caching layer (Redis)
- With 95% cache hit rate: 462K ร— 5% = 23K DB reads โ†’ 1 primary + replicas
- Write load: 9.2K QPS โ†’ 1 primary with async replication
graph TD A[Client: 462K read QPS] --> B[Load Balancer] B --> C[App Servers: 70x] C --> D{Redis Cache
95% hit rate} D -->|Cache Hit: 439K| E[Return from Cache] D -->|Cache Miss: 23K| F[PostgreSQL Primary] F --> G[Read Replicas: 5x] H[Client: 9.2K write QPS] --> I[Load Balancer] I --> J[App Servers: 2x] J --> K[PostgreSQL Primary] K --> L[Async Replication] L --> G style D fill:#fbc02d style F fill:#1976d2

Common Estimation Mistakes

  • Using average QPS instead of peak: Peak can be 2-10x average (Black Friday, breaking news)
  • Ignoring replication: Storage needs 3x for durability, servers need N+2 for availability
  • Forgetting metadata: A "280 char tweet" is actually 1-2 KB with user_id, timestamp, etc.
  • Not accounting for growth: Design for 10x growth over 3 years
  • Assuming even distribution: 80/20 rule applies (20% of users generate 80% of content)
Interview Tip: Always state your assumptions out loud. "I'm assuming 200M DAU, but if it's higher, we'd scale linearly by adding more servers." Shows you understand the trade-offs.

Tail Latency (P99, P999)

Average latency is a lie. If 99% of requests finish in 10ms but 1% take 5 seconds, your users have a terrible experience.

Percentiles Defined

Percentile Meaning Example (if P99 = 100ms)
P50 (Median) 50% of requests faster than this P50 might be 20ms
P95 95% of requests faster than this P95 might be 60ms
P99 99% of requests faster than this 100ms (given)
P999 (P99.9) 99.9% of requests faster than this P999 might be 500ms
%%{init: {'theme':'dark'}}%% graph LR A[1000 Requests] --> B[P50: 500 requests < 20ms] A --> C[P95: 950 requests < 60ms] A --> D[P99: 990 requests < 100ms] A --> E[P999: 999 requests < 500ms] A --> F[1 request: 5 seconds] style B fill:#2e7d32 style C fill:#66bb6a style D fill:#fbc02d style E fill:#f57c00 style F fill:#c62828
Why P99 Matters More Than Average:

If you have 1M requests/day and P99 = 1 second:
โ†’ 1% of requests (10,000) take โ‰ฅ1 second
โ†’ That's 10,000 frustrated users/day
โ†’ Average might be 50ms (looks great!) but UX is bad

Tail Latency Amplification

The real danger: fan-out. When one request triggers multiple backend calls, tail latency multiplies.

sequenceDiagram participant User participant Gateway participant Service1 participant Service2 participant Service3 participant Service4 User->>Gateway: Request par Parallel fan-out Gateway->>Service1: Call 1 Gateway->>Service2: Call 2 Gateway->>Service3: Call 3 Gateway->>Service4: Call 4 end Note over Service1: P99 = 100ms
Hits slow path (1% chance) Service1-->>Gateway: 1000ms (slow!) Service2-->>Gateway: 20ms Service3-->>Gateway: 25ms Service4-->>Gateway: 30ms Note over Gateway: Waits for slowest (1000ms) Gateway-->>User: 1000ms total

Math:

If each service has P99 = 100ms (1% of requests slow):

Probability all 4 services are fast = 0.99^4 = 0.96

โ†’ 4% of requests hit tail latency (4x worse than 1%)

With 10 services: 0.99^10 = 0.90
โ†’ 10% of requests hit tail latency (10x worse!)

This is why microservices can have terrible P99.
Interview Tip: When asked "how do we reduce tail latency?", mention:
1. Hedged requests: Send duplicate request after P99 timeout, use whichever responds first
2. Reduce fan-out: Batch calls, use caching, denormalize data
3. Timeouts: Fail fast rather than waiting 30 seconds
4. Load balancing: Avoid hot spots (power-of-two-choices algorithm)

Service Level Objectives (SLOs)

SLOs define acceptable performance using percentiles:

Service SLO Meaning
API Gateway P99 < 200ms 99% of requests finish under 200ms
Database P95 < 10ms 95% of queries under 10ms
Search P50 < 50ms, P99 < 500ms Median fast, tail acceptable
Batch Job P95 < 1 hour 95% of jobs finish in 1 hour

Measuring Percentiles

Never use averages in production monitoring. Track histograms:

# Prometheus example
http_request_duration_seconds_bucket{le="0.01"} 5000   # 5000 requests < 10ms
http_request_duration_seconds_bucket{le="0.05"} 9500   # 9500 requests < 50ms
http_request_duration_seconds_bucket{le="0.1"} 9900    # 9900 requests < 100ms
http_request_duration_seconds_bucket{le="0.5"} 9990    # 9990 requests < 500ms
http_request_duration_seconds_bucket{le="+Inf"} 10000  # Total requests

P50 โ‰ˆ 10-50ms range
P99 โ‰ˆ 100ms
P999 โ‰ˆ 500ms
%%{init: {'theme':'dark'}}%% xychart-beta title "Latency Distribution (Tail Latency)" x-axis ["P50", "P75", "P90", "P95", "P99", "P999"] y-axis "Latency (ms)" 0 --> 1000 bar [20, 40, 70, 120, 300, 900]

Real-World Example: Amazon

Amazon found that 100ms of additional latency cost them 1% in sales. They optimize for P99.9:

  • Problem: Garbage collection pauses caused P99.9 spikes (5+ seconds)
  • Solution: Move to languages with predictable GC (Java โ†’ Rust for critical paths)
  • Result: P99.9 dropped from 5s to 200ms

Bandwidth-Delay Product (BDP)

The Bandwidth-Delay Product determines how much data can be "in flight" on a network connection before you get acknowledgment.

BDP Formula:

BDP (bytes) = Bandwidth (bytes/sec) ร— RTT (seconds)

Why it matters: TCP window size must be โ‰ฅ BDP to fully utilize bandwidth.

Worked Example: Satellite Link

Satellite connection:
- Bandwidth: 10 Mbps = 1.25 MB/s
- RTT: 500ms (geostationary orbit)

BDP = 1.25 MB/s ร— 0.5s = 625 KB

This means 625 KB of data is "in flight" at any moment.

If TCP window size < 625 KB:
โ†’ Sender must wait for ACKs
โ†’ Link is underutilized
โ†’ Effective throughput < 10 Mbps

If TCP window size โ‰ฅ 625 KB:
โ†’ Sender can keep transmitting
โ†’ Link is fully utilized
sequenceDiagram participant Sender participant Network participant Receiver Note over Sender,Receiver: BDP = 625 KB (500ms ร— 1.25 MB/s) Sender->>Network: Send 625 KB (window size) Note over Network: Data in flight (500ms RTT) Network->>Receiver: Deliver data Receiver->>Network: ACK Network->>Sender: ACK arrives (500ms later) Note over Sender: Can now send next 625 KB alt Window size < BDP Note over Sender: Must wait for ACK
Link underutilized else Window size >= BDP Note over Sender: Can keep sending
Link fully utilized end

TCP Window Scaling

TCP header limits window size to 65,535 bytes (16-bit field). Too small for high-bandwidth or high-latency links!

Link Bandwidth RTT BDP Problem?
LAN 1 Gbps 1ms 125 KB No (< 65 KB)
Cross-country 10 Gbps 50ms 62.5 MB Yes! (need window scaling)
Satellite 10 Mbps 500ms 625 KB Yes! (need window scaling)

Solution: TCP Window Scaling (RFC 1323) multiplies window size by up to 2^14 = 16,384:

Max window = 65,535 ร— 16,384 = 1 GB

Now high-BDP links work:
- 10 Gbps ร— 50ms = 62.5 MB < 1 GB โœ“

Implications for System Design

Interview Tip: When asked about cross-region replication:
"Even with 10 Gbps bandwidth, 100ms RTT means we need large TCP windows and careful tuning. Bulk transfers will be slower than intra-region. Consider multi-region writes with async replication instead of synchronous."

Buffer Sizing in Routers

Rule of thumb: Router buffer = BDP of the link

10 Gbps link, 10ms RTT:
BDP = 1.25 GB/s ร— 0.01s = 12.5 MB

Router buffer should be ~12.5 MB to absorb bursts.

Too small: Packet loss under load
Too large: Bufferbloat (high latency)
graph TD A[Data arrives in bursts] --> B{Buffer Size} B -->|Too Small| C[Packets dropped
Retransmissions
Low throughput] B -->|Just Right: ~BDP| D[Smooth traffic
Low latency
High throughput] B -->|Too Large| E[Bufferbloat
High latency
Jitter] style C fill:#c62828 style D fill:#2e7d32 style E fill:#f57c00

Amdahl's Law & Universal Scalability Law

Adding more CPUs or servers doesn't always make things faster. These laws explain why.

Amdahl's Law

Amdahl's Law Formula:

Speedup = 1 / (S + (1-S)/N)

Where:
S = fraction of program that's serial (can't parallelize)
N = number of processors
1-S = fraction that's parallel

Worked Example:

Video encoding pipeline:
- 5% serial (file I/O, setup): S = 0.05
- 95% parallel (encoding frames): 1-S = 0.95

With 1 CPU: Speedup = 1
With 2 CPUs: Speedup = 1 / (0.05 + 0.95/2) = 1.90  (90% faster)
With 4 CPUs: Speedup = 1 / (0.05 + 0.95/4) = 3.48  (248% faster)
With 10 CPUs: Speedup = 1 / (0.05 + 0.95/10) = 6.90
With 100 CPUs: Speedup = 1 / (0.05 + 0.95/100) = 16.8
With โˆž CPUs: Speedup = 1 / 0.05 = 20 (max possible)

Diminishing returns: 10 CPUs gets you 69% of max speedup.
%%{init: {'theme':'dark'}}%% xychart-beta title "Amdahl's Law: Speedup vs Processors" x-axis [1, 2, 4, 8, 16, 32, 64] y-axis "Speedup" 0 --> 20 line [1, 1.9, 3.5, 6.4, 10.7, 15.0, 17.7]
Interview Tip: When asked "can we scale this horizontally?", identify the serial bottleneck:
- Database writes (single primary)
- Leader election
- Global locks
- Sequential file processing

"Even with 100 servers, the serial portion limits total speedup."

Universal Scalability Law (USL)

Amdahl assumes perfect parallel execution. In reality, coordination overhead grows with worker count.

USL Formula:

Speedup = N / (1 + ฮฑ(N-1) + ฮฒN(N-1))

Where:
N = number of processors
ฮฑ (alpha) = contention coefficient (waiting for locks)
ฮฒ (beta) = coherency coefficient (cache invalidation, data sync)

Example: Database with Locking

Database cluster:
- ฮฑ = 0.05 (5% contention from row locks)
- ฮฒ = 0.02 (2% coherency cost from cache sync)

With 1 node: Speedup = 1
With 2 nodes: Speedup = 2 / (1 + 0.05ร—1 + 0.02ร—2ร—1) = 1.85
With 4 nodes: Speedup = 4 / (1 + 0.05ร—3 + 0.02ร—4ร—3) = 3.17
With 10 nodes: Speedup = 10 / (1 + 0.05ร—9 + 0.02ร—10ร—9) = 5.52
With 20 nodes: Speedup = 20 / (1 + 0.05ร—19 + 0.02ร—20ร—19) = 6.33

Peak at ~15 nodes, then speedup degrades!
%%{init: {'theme':'dark'}}%% xychart-beta title "Universal Scalability Law: Speedup Peaks Then Declines" x-axis [1, 2, 4, 8, 16, 32, 64] y-axis "Speedup" 0 --> 10 line [1, 1.85, 3.17, 5.2, 6.5, 5.8, 3.9]

Comparison: Amdahl vs USL

Aspect Amdahl's Law Universal Scalability Law
Accounts for Serial portion only Serial + contention + coherency
Max speedup 1/S (finite) Can peak then decline
Shape Monotonically increasing, asymptotic Inverted U (peak then drop)
Realistic? Optimistic Matches real-world systems
Interview Tip: USL explains why "just add more servers" eventually makes things worse:
- More cache invalidation traffic
- More coordination overhead (consensus protocols)
- More cross-region replication lag

"Beyond 20 nodes, we'd need sharding to break the contention bottleneck."

Real-World Examples

  • PostgreSQL: Read replicas scale well (low ฮฑ, ฮฒ). Write throughput hits primary bottleneck (Amdahl).
  • Cassandra: Designed for low ฮฑ, ฮฒ (no coordination). Scales near-linearly to hundreds of nodes.
  • Kafka: Partition-level parallelism. Each partition is serial but many partitions scale well.
  • Redis Cluster: Gossip protocol overhead (ฮฒ) grows with cluster size. Practical limit ~100 nodes.

Little's Law

Little's Law relates three metrics: items in system, arrival rate, and time spent. It's the most useful formula in capacity planning.

Little's Law:

L = ฮป ร— W

Where:
L = average number of items in the system
ฮป (lambda) = average arrival rate
W = average time an item spends in the system

If you know any two, you can solve for the third.

Example 1: Connection Pool Sizing

Web server connecting to database:
- ฮป = 1000 requests/sec
- W = 50ms average query time = 0.05 sec

How many DB connections needed?

L = ฮป ร— W
L = 1000 ร— 0.05
L = 50 connections

Add 20% headroom โ†’ 60 connections in the pool.
graph LR A[Requests: 1000/sec] --> B[Connection Pool] B --> C[DB Connection 1
50ms avg] B --> D[DB Connection 2
50ms avg] B --> E[...] B --> F[DB Connection 50
50ms avg] style B fill:#1976d2 style C fill:#66bb6a style D fill:#66bb6a style F fill:#66bb6a
Interview Tip: If pool is too small (L < ฮป ร— W), requests queue up and latency increases. If too large, wasted connections. Little's Law gives you the exact size needed.

Example 2: Queue Capacity

Message queue (Kafka, RabbitMQ):
- ฮป = 10,000 messages/sec
- W = 5 seconds (consumer processing time)

How many messages are in the queue at any moment?

L = 10,000 ร— 5 = 50,000 messages

If queue capacity < 50,000:
โ†’ Messages will be dropped (overflow)

If queue capacity = 100,000:
โ†’ 2x headroom for bursts โœ“

Example 3: Thread Pool Sizing

Background job processor:
- ฮป = 500 jobs/sec
- W = 200ms average job duration = 0.2 sec

How many worker threads needed?

L = 500 ร— 0.2 = 100 threads

With 100 threads:
โ†’ Each thread handles 5 jobs/sec
โ†’ Total throughput = 100 ร— 5 = 500 jobs/sec โœ“
sequenceDiagram participant Queue as Job Queue (ฮป=500/sec) participant T1 as Thread 1 participant T2 as Thread 2 participant T100 as Thread 100 Queue->>T1: Job (W=200ms) Queue->>T2: Job (W=200ms) Queue->>T100: Job (W=200ms) Note over T1,T100: L = 100 threads needed
to handle 500 jobs/sec T1-->>Queue: Done T2-->>Queue: Done T100-->>Queue: Done

Example 4: Reverse Calculation (SLA Enforcement)

SLA requirement: Max 100 concurrent requests in system
Arrival rate: ฮป = 2000 requests/sec

What's the max allowed latency?

W = L / ฮป
W = 100 / 2000
W = 0.05 sec = 50ms

To meet the concurrency SLA, you must keep latency under 50ms.
If latency increases to 100ms:
โ†’ L = 2000 ร— 0.1 = 200 concurrent requests (SLA violated)
Practical Uses of Little's Law:
1. Connection pool sizing: ฮป ร— W = connections needed
2. Queue capacity: ฮป ร— W = buffer size
3. Thread pool sizing: ฮป ร— W = threads needed
4. Latency targets: W = L / ฮป (for fixed concurrency)
5. System limits: L = ฮป ร— W (how many items in flight)

Common Mistakes

  • Using peak vs average: Use peak ฮป and average W for safety
  • Forgetting variance: If W has high variance, add more headroom (2x instead of 1.2x)
  • Ignoring queueing delay: W includes queueing time + processing time
Interview Tip: Little's Law applies to any stable system:
- Airport security (people in line = arrival rate ร— wait time)
- Hospital ER (patients = arrival rate ร— treatment time)
- Data pipeline (records = ingestion rate ร— processing time)

Use it to reason about capacity at every layer.

Powers of 2

Memorize these for instant mental math in interviews.

Power Exact Value Approx (Base 10) Mnemonic
2^10 1,024 ~10^3 (thousand) Kilo (KB)
2^20 1,048,576 ~10^6 (million) Mega (MB)
2^30 1,073,741,824 ~10^9 (billion) Giga (GB)
2^40 1,099,511,627,776 ~10^12 (trillion) Tera (TB)
2^50 ~1.126 ร— 10^15 ~10^15 (quadrillion) Peta (PB)

Quick Conversions

# Storage
1 KB = 2^10 bytes โ‰ˆ 1 thousand
1 MB = 2^20 bytes โ‰ˆ 1 million
1 GB = 2^30 bytes โ‰ˆ 1 billion
1 TB = 2^40 bytes โ‰ˆ 1 trillion

# Time (seconds in...)
1 minute = 60 sec โ‰ˆ 2^6
1 hour = 3,600 sec โ‰ˆ 2^12
1 day = 86,400 sec โ‰ˆ 2^17 (actually 2^16.4)
1 year โ‰ˆ 31.5M sec โ‰ˆ 2^25

# Network
1 Gbps = 125 MB/s = 2^27 bytes/sec (approx)
10 Gbps = 1.25 GB/s = 2^30 bytes/sec (approx)

Mental Math Tricks

Multiplying Powers:
2^a ร— 2^b = 2^(a+b)

Example: 1 KB ร— 1 MB = 2^10 ร— 2^20 = 2^30 = 1 GB
Dividing Powers:
2^a / 2^b = 2^(a-b)

Example: 1 GB / 1 KB = 2^30 / 2^10 = 2^20 = 1 million

Interview Example:

Q: "How many 1 KB records fit in 1 TB?"

A: 1 TB / 1 KB
   = 2^40 / 2^10
   = 2^30
   = ~1 billion records

Q: "1 million users upload 3 MB photos. Total storage?"

A: 10^6 users ร— 3 MB
   โ‰ˆ 2^20 ร— 3 ร— 2^20
   = 3 ร— 2^40
   = 3 TB

Common Interview Numbers

Scenario Powers of 2 Result
1M users ร— 1KB profile 2^20 ร— 2^10 2^30 = 1 GB
1B tweets ร— 1KB each 2^30 ร— 2^10 2^40 = 1 TB
100M photos ร— 3MB each ~2^27 ร— 3 ร— 2^20 3 ร— 2^47 โ‰ˆ 384 TB
Requests/year โ†’ req/sec Total / 2^25 (Year = 31.5M sec โ‰ˆ 2^25)
Interview Tip: Practice converting on the fly:
- "10 million users" โ†’ "that's about 2^23"
- "5 MB photo" โ†’ "call it 2^22 bytes"
- "100K requests/sec" โ†’ "roughly 2^17"

Rounding makes math trivial.

Powers of 10 vs Powers of 2

Base 10 Closest Power of 2 Error
1K (10^3) 1,024 (2^10) +2.4%
1M (10^6) 1,048,576 (2^20) +4.9%
1B (10^9) 1,073,741,824 (2^30) +7.4%
1T (10^12) 1,099,511,627,776 (2^40) +10%

Error compounds but stays under 10%. Perfectly acceptable for interviews.

Capacity Planning: Worked Examples

Let's apply everything to real system design questions.

Example 1: Design Twitter

Requirements:

  • 300M daily active users (DAU)
  • Each user views 100 tweets/day (read)
  • Each user posts 2 tweets/day (write)
  • Tweet size: 280 chars ร— 2 bytes + 200 bytes metadata = ~1 KB
  • Photos: 20% of tweets include one 2MB photo
  • Retention: 5 years

Step 1: Traffic Estimation

Read QPS:
- 300M users ร— 100 reads/day = 30B reads/day
- 30B / 86,400 sec = 347K reads/sec
- Peak (2x): 694K reads/sec

Write QPS:
- 300M users ร— 2 tweets/day = 600M tweets/day
- 600M / 86,400 = 6,944 tweets/sec
- Peak (2x): ~14K writes/sec

Step 2: Storage Estimation

Text storage:
- 600M tweets/day ร— 1 KB = 600 GB/day
- 5 years = 600 GB ร— 365 ร— 5 = 1.1 PB

Photo storage:
- 20% of tweets have photos: 600M ร— 0.2 = 120M photos/day
- 120M ร— 2 MB = 240 TB/day
- 5 years = 240 TB ร— 365 ร— 5 = 438 PB

Total: 1.1 PB + 438 PB โ‰ˆ 440 PB (photos dominate)

With 3x replication: 1.3 EB

Step 3: Bandwidth Estimation

Read bandwidth (timeline loads):
- Each timeline request returns 20 tweets
- 20 tweets ร— 1 KB = 20 KB (text)
- 20% have photos: 4 tweets ร— 2 MB = 8 MB (photos via CDN)
- Text bandwidth: 694K QPS ร— 20 KB / 125 = 111 Gbps
- Photo bandwidth: 694K QPS ร— 8 MB / 125 = 44 Tbps (CDN serves this)

Write bandwidth:
- 14K tweets/sec ร— 1 KB = 14 MB/sec (negligible)
- Photo uploads: 14K ร— 0.2 ร— 2 MB = 5.6 GB/sec = 45 Gbps

Step 4: Server Count

API servers (assume 10K QPS per server after caching):
- Read: 694K / 10K = 70 servers
- Write: 14K / 10K = 2 servers
- Total: ~72 API servers (with load balancing)

Database:
- Write load (14K tweets/sec) โ†’ 1 primary + async replicas
- Read load (694K QPS) โ†’ needs massive caching
- Cache hit rate: 90% (popular tweets cached)
- DB reads: 694K ร— 10% = 69K QPS โ†’ 5-10 read replicas

Cache (Redis cluster):
- Store hot tweets (last 24 hours)
- 600M tweets/day ร— 1 KB = 600 GB
- Add 2x for working set: 1.2 TB
- Redis cluster (64 GB per node): 20 nodes

Storage:
- Text: PostgreSQL (sharded by user_id)
- Photos: S3 (with CloudFront CDN)
graph TD A[Client: 694K read QPS] --> B[Load Balancer] B --> C[API Servers: 70x] C --> D[Redis Cache: 20 nodes
90% hit rate] D -->|Hit: 624K| E[Return Cached] D -->|Miss: 69K| F[PostgreSQL Primary] F --> G[Read Replicas: 10x] H[Client: 14K write QPS] --> I[API Servers: 2x] I --> J[PostgreSQL Primary
Sharded by user_id] J --> K[Async Replication] K --> G I --> L[S3: Photos
438 PB] L --> M[CloudFront CDN
44 Tbps] style D fill:#fbc02d style F fill:#1976d2 style L fill:#43a047
Interview Tip: Always mention:
- Caching strategy: "90% hit rate from Redis reduces DB load 10x"
- CDN for photos: "S3 + CloudFront handles 44 Tbps without hitting origin"
- Sharding: "Shard tweets by user_id for horizontal scaling"
- Denormalization: "Store tweet text in timeline cache (no join needed)"

Example 2: Design YouTube

Requirements:

  • 2B monthly active users (MAU), 500M DAU
  • Each user watches 5 videos/day (30 min total)
  • Each user uploads 0.01 videos/day (1 upload per 100 users/day)
  • Average video: 10 minutes, 1080p, 150 MB (after compression)
  • Transcode to 4 qualities: 360p (50 MB), 720p (100 MB), 1080p (150 MB), 4K (400 MB)

Step 1: Traffic Estimation

Video views:
- 500M DAU ร— 5 videos/day = 2.5B views/day
- 2.5B / 86,400 = 28,935 views/sec
- Peak (3x for prime time): ~87K views/sec

Video uploads:
- 500M DAU ร— 0.01 = 5M uploads/day
- 5M / 86,400 = 58 uploads/sec

Step 2: Storage Estimation

Upload storage (original):
- 5M videos/day ร— 150 MB = 750 TB/day

Transcoded storage (all qualities):
- Per video: 50 + 100 + 150 + 400 = 700 MB
- 5M videos/day ร— 700 MB = 3.5 PB/day
- 5 years: 3.5 PB ร— 365 ร— 5 = 6.4 EB

With 3x replication: 19.2 EB

Step 3: Bandwidth Estimation

Video streaming (peak):
- 87K views/sec
- Assume 50% watch 1080p (150 MB per 10 min)
- 150 MB / 600 sec = 250 KB/sec per stream
- 87K ร— 250 KB/sec = 21.75 GB/sec = 174 Gbps

CDN serves this (not origin servers).

Upload bandwidth:
- 58 uploads/sec ร— 150 MB = 8.7 GB/sec = 70 Gbps

Step 4: Compute (Transcoding)

Transcoding cost:
- 5M videos/day
- 10 minutes of video โ†’ 30 minutes of compute (transcode to 3 formats)
- 5M ร— 30 min = 150M minutes/day = 104K hours/day

AWS Elastic Transcoder: $0.03 per minute (rough estimate)
- 150M minutes/day ร— $0.03 = $4.5M/day = $1.64B/year

Cheaper to build own transcoding farm:
- 104K hours/day / 24 hours = 4,333 servers needed
- AWS c5.4xlarge (16 vCPU, $0.68/hour): $4.3M/month = $51.6M/year
- 70% savings vs managed service

Step 5: Database

Metadata:
- 5M videos/day ร— 5 KB metadata = 25 GB/day (negligible)
- 5 years: 45 TB (easily fits in PostgreSQL)

Queries:
- Video metadata fetch: 87K QPS (read-heavy)
- Use read replicas + cache (Redis)
- Cache popular videos (80/20 rule): top 20% = 40% of views
- 87K ร— 40% = 35K QPS served from cache
- 52K QPS โ†’ database (10 read replicas @ 5K QPS each)
graph TD A[User: 87K views/sec] --> B[Load Balancer] B --> C[API Servers: 20x] C --> D[Redis Cache
Popular videos 40%] D -->|Hit: 35K| E[Return Metadata] D -->|Miss: 52K| F[PostgreSQL] F --> G[Read Replicas: 10x] C --> H[CDN: CloudFront] H --> I[S3: Video Storage
19.2 EB] J[Upload: 58/sec] --> K[Upload API: 5x] K --> L[S3: Raw Videos] L --> M[Transcoding Queue
SQS] M --> N[Transcode Workers: 4333x] N --> I style D fill:#fbc02d style H fill:#43a047 style N fill:#f57c00

Example 3: Design WhatsApp

Requirements:

  • 2B daily active users
  • Each user sends 50 messages/day
  • Average message size: 100 bytes (text only for simplicity)
  • Real-time delivery (WebSocket connections)
  • Retention: 30 days (messages deleted after)

Step 1: Traffic Estimation

Messages per day:
- 2B users ร— 50 messages = 100B messages/day
- 100B / 86,400 = 1.16M messages/sec
- Peak (2x): 2.3M messages/sec

Step 2: Storage Estimation

Daily storage:
- 100B messages ร— 100 bytes = 10 TB/day

30-day retention:
- 10 TB ร— 30 = 300 TB
- With 3x replication: 900 TB โ‰ˆ 1 PB

(Much smaller than Twitter/YouTube because messages are tiny and ephemeral)

Step 3: Connection Count

WebSocket connections:
- 2B DAU, assume 50% online concurrently during peak: 1B connections

Connections per server:
- Linux: ~10K connections per server (with tuning, can push to 50K-100K)
- Use 50K per server: 1B / 50K = 20,000 servers

These are lightweight "connection servers" (just hold WebSocket, minimal logic).

Step 4: Message Delivery

Delivery architecture:
1. User A sends message โ†’ hits API server
2. API server writes to database
3. API server publishes to message queue (Kafka)
4. Connection server for User B (recipient) subscribed to Kafka
5. Connection server pushes message over WebSocket to User B

Kafka throughput:
- 2.3M messages/sec
- Kafka can handle 1M+ messages/sec per broker
- 3-5 brokers for redundancy and throughput
sequenceDiagram participant A as User A participant API as API Server participant DB as Cassandra participant Kafka as Kafka participant ConnB as Conn Server (User B) participant B as User B A->>API: Send message (2.3M/sec peak) API->>DB: Write message API->>Kafka: Publish to topic Kafka->>ConnB: Subscribe to topic ConnB->>B: Push via WebSocket Note over ConnB: 20K servers
50K connections each

Step 5: Database Choice

Requirements:
- 2.3M writes/sec (very high write load)
- 30-day retention (time-series data)
- Eventual consistency OK (message delivery โ‰  banking)

Choice: Cassandra (write-optimized, scales horizontally)

Capacity:
- 300 TB storage
- Cassandra rule: 1 TB per node
- 300 nodes (with RF=3, raw 100 nodes)

Write throughput:
- Cassandra: ~100K writes/sec per node
- 2.3M writes/sec / 100K = 23 nodes (well within 300 node cluster)
Interview Tip: WhatsApp's real architecture uses:
- Erlang servers for connection handling (2M+ connections per server!)
- FreeBSD for optimized networking
- Mnesia + custom storage instead of Cassandra

But Cassandra is a great interview answer (widely known, fits requirements).

Latency Comparison at Scale

Let's visualize the full request path and where latency comes from.

Full Request Flow

sequenceDiagram participant User participant CDN participant LB as Load Balancer participant API as API Server participant Cache as Redis Cache participant DB as Database participant Disk User->>CDN: Request (150ms cross-continent) Note over CDN: CDN miss CDN->>LB: Forward (500ฮผs intra-DC) LB->>API: Route (100ฮผs) Note over API: CPU processing (1ms) API->>Cache: Check cache (500ฮผs network + 100ฮผs Redis) Note over Cache: Cache miss Cache-->>API: Not found API->>DB: Query (500ฮผs network + 5ms DB query) DB->>Disk: Read from SSD (100ฮผs) Disk-->>DB: Data DB-->>API: Result (500ฮผs network) API-->>LB: Response (100ฮผs) LB-->>CDN: Response (500ฮผs) CDN-->>User: Response (150ms) Note over User,Disk: Total: ~157ms (dominated by network)

Latency Budget Breakdown

Hop Latency % of Total Optimization
User โ†’ CDN (cross-continent) 150 ms 95.5% Multi-region CDN, edge compute
CDN โ†’ Load Balancer 0.5 ms 0.3% Co-locate CDN POPs with DC
Load Balancer โ†’ API 0.1 ms 0.06% Low-latency LB (Envoy, HAProxy)
API processing 1 ms 0.6% Efficient code, avoid GC pauses
API โ†’ Redis (miss) 0.6 ms 0.4% Co-locate cache with API
API โ†’ DB 0.5 ms 0.3% Co-locate DB with API
DB query + SSD read 5.1 ms 3.2% Optimize query, use indexes, NVMe

Key Insight: 95% of latency is network (user to CDN). Optimizing API or DB from 5ms to 1ms saves 4ms (2.5% improvement). Adding a nearby CDN PoP saves 50ms (32% improvement).

Caching Impact

Scenario: 90% cache hit rate

Without cache:
- Every request hits DB (5ms DB latency)
- Total latency: 150ms + 1ms + 5ms = 156ms (average)

With cache (90% hit rate):
- 90% of requests: 150ms + 1ms + 0.1ms (Redis) = 151ms
- 10% of requests: 150ms + 1ms + 5ms (DB miss) = 156ms
- Average: 0.9 ร— 151 + 0.1 ร— 156 = 151.4ms

Savings: 4.6ms (3% faster)

This seems small, but at 1M QPS:
- Without cache: DB handles 1M queries/sec (impossible)
- With cache: DB handles 100K queries/sec (feasible with replicas)
Interview Tip: Caching doesn't dramatically reduce latency (5ms โ†’ 0.1ms on 150ms budget = 3%). But it does dramatically reduce load (10x fewer DB queries). This lets you handle 10x more traffic with same DB capacity.

When Caching Pays Off

Caching ROI Formula:

Latency Saved = Hit Rate ร— (DB Latency - Cache Latency)
Load Reduction = Hit Rate ร— 100%

Example: 90% hit rate, 5ms DB, 0.1ms cache
โ†’ Latency saved: 0.9 ร— 4.9ms = 4.4ms
โ†’ Load reduction: 90% (DB sees 10% of traffic)
%%{init: {'theme':'dark'}}%% xychart-beta title "Cache Hit Rate vs DB Load Reduction" x-axis ["0%", "50%", "70%", "90%", "95%", "99%"] y-axis "DB Load (% of original)" 0 --> 100 line [100, 50, 30, 10, 5, 1]

Latency at Different Scales

System Scale Dominant Latency
Single-server app 1K QPS DB query (5-50ms)
Regional app 10K QPS Intra-region network (1-5ms)
Global app (CDN) 100K+ QPS User โ†’ CDN (50-200ms)
Microservices (10+ calls) Any scale Service-to-service (10-100ms cumulative)
Mobile app (3G) N/A Network (200-500ms)

Network Latency by Geography

Physical distance matters. Speed of light imposes a lower bound on latency.

Speed of Light Limits

Speed of light in fiber: ~200,000 km/sec (slower than vacuum due to refractive index)

New York to San Francisco: 4,000 km
- Minimum RTT: (4,000 km ร— 2) / 200,000 km/sec = 40ms

New York to London: 5,500 km
- Minimum RTT: (5,500 ร— 2) / 200,000 = 55ms

New York to Tokyo: 11,000 km
- Minimum RTT: (11,000 ร— 2) / 200,000 = 110ms

Sydney to London: 17,000 km
- Minimum RTT: (17,000 ร— 2) / 200,000 = 170ms

You can't beat physics. No optimization gets below these limits.

Real-World Latencies

Route Distance Theoretical Min RTT Actual RTT
Same datacenter 0.1 km 1 ฮผs 500 ฮผs (0.5ms)
Same city (NYC to NYC) 10 km 100 ฮผs 1-2 ms
Same region (US East) 500 km 5 ms 10-20 ms
US East to US West 4,000 km 40 ms 60-80 ms
US to Europe 6,000 km 60 ms 80-120 ms
US to Asia 11,000 km 110 ms 150-200 ms
Europe to Australia 17,000 km 170 ms 250-300 ms

Why actual > theoretical?

  • Fiber doesn't take shortest path (follows roads, goes around mountains)
  • Multiple hops through routers (each adds 0.1-1ms)
  • Queuing delays in routers during congestion
  • Protocol overhead (TCP handshake, TLS negotiation)
graph TD A[New York] -->|60-80ms| B[San Francisco] A -->|80-120ms| C[London] A -->|150-200ms| D[Tokyo] B -->|100-150ms| C B -->|110-160ms| D C -->|250-300ms| E[Sydney] D -->|110-150ms| E style A fill:#1976d2 style B fill:#1976d2 style C fill:#43a047 style D fill:#f57c00 style E fill:#e64a19

Implications for System Design

1. Multi-Region Replication is Slow

Synchronous replication (wait for ACK before commit):
- US East โ†’ US West: 80ms added to every write
- US โ†’ Europe: 120ms added
- US โ†’ Asia: 200ms added

This is why global databases use:
- Async replication (eventual consistency)
- Or CRDT (conflict-free replicated data types)
- Or Raft/Paxos with regional leaders

2. CDN is Essential for Global Apps

User in Sydney requests page from US origin:
- 250ms RTT ร— 3 round trips (DNS, TCP, TLS) = 750ms
- Before any data transfers!

User in Sydney requests page from Sydney CDN PoP:
- 1ms RTT ร— 3 = 3ms
- 250x faster

3. Chatty Protocols Fail Globally

API that requires 10 sequential requests:
- Intra-datacenter: 10 ร— 0.5ms = 5ms
- US to Asia: 10 ร— 180ms = 1,800ms (1.8 seconds!)

Solution: Batch requests, use GraphQL, or switch to async messaging.
Interview Tip: When designing global systems, mention:
1. CDN for static content: "Serve from nearest PoP"
2. Multi-region deployment: "Deploy API servers in US, EU, Asia"
3. Regional databases: "Users in Asia hit Asia DB, eventual consistency via async replication"
4. Async where possible: "Analytics, notifications, email can be async"
5. Reduce round trips: "Batch API calls, use HTTP/2 multiplexing"

AWS Region Latency Matrix

From \ To us-east-1 us-west-2 eu-west-1 ap-southeast-1
us-east-1 - 70ms 90ms 220ms
us-west-2 70ms - 140ms 160ms
eu-west-1 90ms 140ms - 170ms
ap-southeast-1 220ms 160ms 170ms -

Design Patterns for Global Systems

Pattern 1: Active-Active (Regional)

graph TD A[User: US East] --> B[US East Region] C[User: Europe] --> D[Europe Region] E[User: Asia] --> F[Asia Region] B --> G[Regional DB: US] D --> H[Regional DB: EU] F --> I[Regional DB: Asia] G <-->|Async replication
eventual consistency| H H <-->|Async replication| I I <-->|Async replication| G style B fill:#1976d2 style D fill:#43a047 style F fill:#f57c00

Pros: Fast local writes (no cross-region RTT)
Cons: Eventual consistency, conflict resolution needed

Pattern 2: Active-Passive (Failover)

graph TD A[All Users] --> B[Primary: US East] B --> C[Primary DB] B -->|Async replication| D[Standby: Europe] D --> E[Standby DB] style B fill:#2e7d32 style D fill:#757575

Pros: Strong consistency (single primary)
Cons: High latency for distant users, single point of failure

Pattern 3: Geo-Partitioned

graph TD A[US Users] --> B[US Region] C[EU Users] --> D[EU Region] B --> E[DB: US users only] D --> F[DB: EU users only] style B fill:#1976d2 style D fill:#43a047

Pros: Fast local access, strong consistency within partition
Cons: Cross-region queries expensive, data residency compliance

Summary: Key Takeaways

  1. Memorize Jeff Dean's numbers: L1 cache (1ns), RAM (100ns), SSD (16ฮผs), network (500ฮผs), cross-continent (150ms). These ratios guide every design decision.
  2. Throughput โ‰  Latency: A 10 Gbps link still has 100ms latency to Asia. High bandwidth doesn't fix high latency.
  3. Tail latency matters more than average: If P99 = 1 sec, 1% of users have a bad experience. With fan-out, tail latency amplifies.
  4. Use Little's Law everywhere: L = ฮป ร— W. Know any two, solve for the third. Essential for connection pools, queues, thread pools.
  5. Amdahl's Law limits parallelism: If 5% is serial, max speedup is 20x even with infinite CPUs. Find and eliminate serial bottlenecks.
  6. USL explains why adding servers can hurt: Coordination overhead grows quadratically. Beyond a certain scale, you need sharding.
  7. Bandwidth-Delay Product sizes buffers: BDP = bandwidth ร— RTT. TCP windows and router buffers must be โ‰ฅ BDP.
  8. Powers of 2 for mental math: 2^10 = KB, 2^20 = MB, 2^30 = GB, 2^40 = TB. Use these to estimate instantly.
  9. Back-of-envelope framework: Users โ†’ QPS โ†’ Storage โ†’ Bandwidth โ†’ Servers. Follow this process in every interview.
  10. Geography is destiny: Speed of light imposes a 170ms minimum RTT from Sydney to London. Multi-region async replication is the only solution.
Final Interview Tip: When doing capacity calculations:
1. State assumptions clearly ("I'm assuming...")
2. Round aggressively (use 100K sec/day instead of 86,400)
3. Show your work (write down the math)
4. Sanity check results ("Does 1 PB make sense for 1M users? Yes, because...")
5. Discuss trade-offs ("We could reduce storage by compressing, but that adds 3ms latency...")

The process matters more than the exact number.
Cheat Sheet for Interviews:

Latency: L1 (1ns), RAM (100ns), SSD (16ฮผs), Network (0.5ms), Cross-continent (150ms)
Storage: 1 KB = 2^10, 1 MB = 2^20, 1 GB = 2^30, 1 TB = 2^40
QPS: QPS = (DAU ร— actions/day) / 100,000
Storage: Daily ร— Retention ร— Replication (usually 3x)
Bandwidth: QPS ร— Response Size / 125 = Gbps
Servers: Peak QPS / (10K QPS per server) ร— 1.5 safety
Little's Law: L = ฮป ร— W (connections = QPS ร— latency)
Cache Hit Rate: 90% hit rate = 10x DB load reduction