Chapter 49: Design WhatsApp
1. Requirements
Functional Requirements
| Feature | Description | Priority |
|---|---|---|
| One-to-One Messaging | Send/receive text messages in real-time | P0 |
| Group Messaging | Support groups up to 1024 members | P0 |
| Media Sharing | Images, videos, audio, documents (max 2GB) | P0 |
| Presence Indicator | Online/offline/last seen status | P0 |
| Delivery Receipts | Sent (1 tick), delivered (2 ticks), read (2 blue ticks) | P0 |
| E2E Encryption | All messages encrypted end-to-end | P0 |
| Offline Support | Queue messages for offline users | P0 |
| Multi-Device Sync | Support up to 4 companion devices | P1 |
Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| Availability | 99.99% (52 min downtime/year) | Critical communication service |
| Latency | <200ms P99 for message delivery | Real-time feel |
| Consistency | Strong per-conversation ordering | Message coherence |
| Scale | 2B users, 100B messages/day | WhatsApp's actual scale |
| Reliability | Zero message loss | Trust requirement |
| Security | E2E encryption, no server-side plaintext | Privacy promise |
Out of Scope
- Voice/video calls (separate service)
- Stories/Status updates
- Payment features
- Business API
2. Capacity Estimation
Traffic Estimates
| Metric | Calculation | Result |
|---|---|---|
| Total Messages/Day | Given | 100B messages |
| Messages/Second (avg) | 100B / 86400s | 1.15M msgs/sec |
| Messages/Second (peak) | 1.15M × 3 (peak factor) | 3.5M msgs/sec |
| Active Users (DAU) | 2B × 0.5 (activity rate) | 1B DAU |
| Concurrent Connections | 1B × 0.05 (concurrency) | 50M concurrent |
| Group Messages | 100B × 0.3 (30% groups) | 30B/day |
| Media Messages | 100B × 0.2 (20% media) | 20B/day |
Storage Estimates
| Data Type | Size | Volume/Day | Total/Day |
|---|---|---|---|
| Text Message Metadata | 500 bytes | 80B messages | 40TB |
| Media Messages | 500KB avg | 20B messages | 10PB |
| User Profiles | 5KB | 2B users | 10TB (one-time) |
| Group Metadata | 10KB | 500M groups | 5TB (one-time) |
| Total/Day | ~10PB | ||
| Retention (18 months) | ~5.5 Exabytes |
Bandwidth Estimates
| Direction | Calculation | Result |
|---|---|---|
| Ingress (text) | 1.15M msgs/s × 500 bytes | 575 MB/s = 4.6 Gbps |
| Ingress (media) | 230K media/s × 500KB | 115 GB/s = 920 Gbps |
| Egress (with fanout) | Ingress × 2 (avg fanout) | ~2 Tbps |
Server Estimates
| Component | Capacity/Server | Servers Needed |
|---|---|---|
| WebSocket Gateway | 2M connections (Erlang) | 25 servers |
| Chat Service | 50K msgs/sec | 70 servers (peak) |
| Cassandra Nodes | 10K writes/sec/node | 350 nodes (peak) |
| Redis (Presence) | 100K ops/sec | 500 instances |
3. High-Level Design
graph TB
subgraph "Client Layer"
C1[Mobile Client A]
C2[Mobile Client B]
C3[Web Client]
end
subgraph "Edge Layer"
LB[Load Balancer]
GW1[WebSocket Gateway 1
Erlang/BEAM] GW2[WebSocket Gateway 2
Erlang/BEAM] GW3[WebSocket Gateway N] end subgraph "Service Layer" CS[Chat Service
Stateless] PS[Presence Service] GS[Group Service] NS[Notification Service
Push] MS[Media Service] AS[Auth Service] end subgraph "Message Queue" KF[Kafka
Message Stream] end subgraph "Storage Layer" CASS[(Cassandra
Message Store)] REDIS[(Redis
Presence/Cache)] PGSQL[(PostgreSQL
User/Group Meta)] S3[(S3
Media Storage)] end subgraph "External" FCM[FCM/APNs
Push Notifications] end C1 & C2 & C3 -->|WSS| LB LB --> GW1 & GW2 & GW3 GW1 & GW2 & GW3 <-->|gRPC| CS GW1 & GW2 & GW3 <-->|gRPC| PS CS --> KF KF --> CS CS --> CASS CS --> PGSQL PS --> REDIS GS --> PGSQL GS --> REDIS CS --> NS NS --> FCM MS --> S3 GW1 & GW2 --> REDIS style GW1 fill:#e1f5ff style GW2 fill:#e1f5ff style GW3 fill:#e1f5ff style KF fill:#fff4e6 style CASS fill:#f3e5f5 style REDIS fill:#ffebee
Erlang/BEAM] GW2[WebSocket Gateway 2
Erlang/BEAM] GW3[WebSocket Gateway N] end subgraph "Service Layer" CS[Chat Service
Stateless] PS[Presence Service] GS[Group Service] NS[Notification Service
Push] MS[Media Service] AS[Auth Service] end subgraph "Message Queue" KF[Kafka
Message Stream] end subgraph "Storage Layer" CASS[(Cassandra
Message Store)] REDIS[(Redis
Presence/Cache)] PGSQL[(PostgreSQL
User/Group Meta)] S3[(S3
Media Storage)] end subgraph "External" FCM[FCM/APNs
Push Notifications] end C1 & C2 & C3 -->|WSS| LB LB --> GW1 & GW2 & GW3 GW1 & GW2 & GW3 <-->|gRPC| CS GW1 & GW2 & GW3 <-->|gRPC| PS CS --> KF KF --> CS CS --> CASS CS --> PGSQL PS --> REDIS GS --> PGSQL GS --> REDIS CS --> NS NS --> FCM MS --> S3 GW1 & GW2 --> REDIS style GW1 fill:#e1f5ff style GW2 fill:#e1f5ff style GW3 fill:#e1f5ff style KF fill:#fff4e6 style CASS fill:#f3e5f5 style REDIS fill:#ffebee
Component Responsibilities
| Component | Responsibility | Technology |
|---|---|---|
| WebSocket Gateway | Maintain persistent connections, route messages to chat service | Erlang/BEAM (2M conn/server) |
| Chat Service | Core message handling, deduplication, storage, forwarding | Go/Java + gRPC |
| Presence Service | Track online/offline status, last seen, typing indicators | Go + Redis |
| Group Service | Group metadata, membership, admin operations | Java + PostgreSQL |
| Notification Service | Send push notifications to offline users | Go + FCM/APNs |
| Media Service | Upload, thumbnail generation, CDN distribution | Go + S3 + ImageMagick |
| Kafka | Message stream, async processing, analytics | Kafka with 7-day retention |
| Cassandra | Message history storage (partition by user_id + chat_id) | Cassandra (AP, write-optimized) |
| Redis | Presence state, session cache, recent messages | Redis Cluster |
| PostgreSQL | User profiles, group metadata, contacts | PostgreSQL (sharded) |
Interview Callout: Gateway Design Choice
Why Erlang/BEAM for gateways? BEAM VM excels at concurrent connections: lightweight processes (2KB/process), preemptive scheduling, built-in supervision trees. WhatsApp achieved 2M+ connections per server. Alternative: Go with epoll can handle ~1M connections/server but needs more careful resource management.
4. Messaging Flow
One-to-One Message Flow
sequenceDiagram
participant A as Client A
participant GW_A as Gateway A
participant CS as Chat Service
participant CASS as Cassandra
participant KF as Kafka
participant REDIS as Redis
participant GW_B as Gateway B
participant B as Client B
participant NS as Notification
A->>GW_A: 1. Send message (WSS)
{msg_id, to_user_id, encrypted_content} GW_A->>CS: 2. Forward via gRPC CS->>CS: 3. Validate & deduplicate
(check msg_id) par Persist Message CS->>CASS: 4a. Store message
partition: (sender_id, chat_id) CS->>CASS: 4b. Store for recipient
partition: (receiver_id, chat_id) and Stream to Analytics CS->>KF: 4c. Publish to Kafka end CS->>GW_A: 5. ACK to sender (sent ✓) GW_A->>A: 6. Sent receipt CS->>REDIS: 7. Lookup recipient gateway REDIS-->>CS: gateway_b:instance_id alt Recipient Online CS->>GW_B: 8. Forward message GW_B->>B: 9. Deliver via WSS B->>GW_B: 10. ACK received GW_B->>CS: 11. Delivery confirmation CS->>GW_A: 12. Delivered receipt (✓✓) GW_A->>A: 13. Update to delivered B->>GW_B: 14. Read receipt (user viewed) GW_B->>CS: 15. Read confirmation CS->>GW_A: 16. Read receipt (✓✓ blue) GW_A->>A: 17. Update to read else Recipient Offline CS->>REDIS: 8. Queue in offline inbox CS->>NS: 9. Trigger push notification NS->>B: 10. Push (FCM/APNs) end
{msg_id, to_user_id, encrypted_content} GW_A->>CS: 2. Forward via gRPC CS->>CS: 3. Validate & deduplicate
(check msg_id) par Persist Message CS->>CASS: 4a. Store message
partition: (sender_id, chat_id) CS->>CASS: 4b. Store for recipient
partition: (receiver_id, chat_id) and Stream to Analytics CS->>KF: 4c. Publish to Kafka end CS->>GW_A: 5. ACK to sender (sent ✓) GW_A->>A: 6. Sent receipt CS->>REDIS: 7. Lookup recipient gateway REDIS-->>CS: gateway_b:instance_id alt Recipient Online CS->>GW_B: 8. Forward message GW_B->>B: 9. Deliver via WSS B->>GW_B: 10. ACK received GW_B->>CS: 11. Delivery confirmation CS->>GW_A: 12. Delivered receipt (✓✓) GW_A->>A: 13. Update to delivered B->>GW_B: 14. Read receipt (user viewed) GW_B->>CS: 15. Read confirmation CS->>GW_A: 16. Read receipt (✓✓ blue) GW_A->>A: 17. Update to read else Recipient Offline CS->>REDIS: 8. Queue in offline inbox CS->>NS: 9. Trigger push notification NS->>B: 10. Push (FCM/APNs) end
Message Schema (Cassandra)
| Column | Type | Description |
|---|---|---|
| user_id | UUID | Partition key (owner of inbox) |
| chat_id | UUID | Clustering key (conversation) |
| message_id | UUID | Clustering key (message identifier) |
| timestamp | TIMESTAMP | Clustering key (DESC order) |
| sender_id | UUID | Who sent the message |
| content | BLOB | Encrypted message payload |
| content_type | TEXT | text/image/video/audio/document |
| status | TEXT | sent/delivered/read |
| media_url | TEXT | S3 URL for media (optional) |
Deduplication Strategy
| Scenario | Solution |
|---|---|
| Client retries same message | Client generates UUID message_id, server checks Redis bloom filter (TTL: 7 days) |
| Network splits during send | Server returns idempotent ACK if msg_id already processed |
| Gateway crashes after forward | Client timeout → retry with same msg_id → server dedupes |
Interview Callout: Why Store Message Twice?
Dual storage: Message stored in both sender's and recipient's partition for efficient queries. Each user's message history query is local to their partition. Trade-off: 2× storage for O(1) read performance. Alternative: single storage with secondary index (slower, requires scatter-gather).
5. Presence Service
Presence States
| State | Trigger | Display |
|---|---|---|
| Online | Active WebSocket connection + heartbeat <30s ago | "online" |
| Away | No heartbeat for 30-300s | "last seen 2 minutes ago" |
| Offline | WebSocket closed OR no heartbeat for >5min | "last seen today at 2:30 PM" |
| Typing | Client sends typing event (rate-limited to 1/3s) | "typing..." |
Heartbeat Protocol
| Aspect | Design | Rationale |
|---|---|---|
| Interval | 30 seconds | Balance battery life vs. presence freshness |
| Timeout | 3 missed heartbeats (90s) → offline | Handle network hiccups gracefully |
| Storage | Redis with 90s TTL | Auto-expire stale presence |
| Piggyback | Any client message = implicit heartbeat | Reduce redundant traffic |
Redis Schema
| Key | Type | Value | TTL |
|---|---|---|---|
| presence:{user_id} | Hash | {status: "online", last_seen: timestamp, gateway: "gw-3"} | 90s |
| typing:{chat_id}:{user_id} | String | "typing" | 5s |
| user_sessions:{user_id} | Set | {device_1, device_2, device_3} | 24h |
Fan-Out Optimization
| Naive Approach | Problem | Optimized Approach |
|---|---|---|
| Broadcast presence to all contacts (500 avg) | 500 updates × 50M concurrent = 25B ops/min | Only notify users with active chat window open |
| Real-time updates every 30s | Unnecessary battery drain | Client subscribes to presence only for visible chats (3-5 typically) |
| Store all contacts' presence per user | 2B users × 500 contacts = 1T entries | Pull-on-demand: fetch presence when chat opened, subscribe for duration |
Presence Update Flow
sequenceDiagram
participant A as Client A
participant GW as Gateway
participant PS as Presence Service
participant REDIS as Redis
participant B as Client B (viewing A's chat)
loop Every 30s
A->>GW: Heartbeat
GW->>PS: Update presence
PS->>REDIS: SET presence:A {online} EX 90
end
B->>GW: Open chat with A
GW->>PS: Subscribe to A's presence
PS->>REDIS: GET presence:A
REDIS-->>PS: {status: online}
PS-->>B: A is online
Note over A,GW: A goes offline (no heartbeat)
REDIS->>REDIS: TTL expires (90s)
PS->>PS: Detect expiry via Redis keyspace notification
PS->>B: A is offline (last seen: timestamp)
Interview Callout: Last Seen Privacy
Privacy controls: Users can hide "last seen" from non-contacts, or entirely. Implementation: presence service checks privacy settings before fan-out. If hidden, return "last seen a long time ago" instead of precise timestamp. Additional optimization: cache privacy settings in Redis with user's presence data.
6. Group Messaging
Group Types
| Type | Size Limit | Architecture | Use Case |
|---|---|---|---|
| Small Group | ≤256 members | Direct fan-out from chat service | Family, friends, team chats |
| Large Group | 257-1024 members | Async fan-out via Kafka + worker pool | Communities, announcements |
| Broadcast Channel | Unlimited subscribers | Pub-sub model, no replies | News, official channels (out of scope) |
Group Metadata (PostgreSQL)
| Table | Columns | Sharding Key |
|---|---|---|
| groups | group_id, name, icon_url, created_at, created_by, member_count | group_id |
| group_members | group_id, user_id, role (admin/member), joined_at | group_id |
| group_settings | group_id, only_admins_send, disappearing_messages_ttl | group_id |
Small Group Message Flow (≤256 members)
sequenceDiagram
participant A as Client A
participant CS as Chat Service
participant CASS as Cassandra
participant REDIS as Redis
participant GW as Gateways
participant M as Members (B, C, D...)
A->>CS: Send group message
CS->>CS: Fetch member list
(cached in Redis) par Store for sender CS->>CASS: Store in A's partition and Store for each member loop For each member CS->>CASS: Store in member's partition
(group_id, msg_id, seq_num) end end CS->>REDIS: Get online members & gateways par Fan-out to online members CS->>GW: Forward to member B CS->>GW: Forward to member C CS->>GW: Forward to member D Note over CS,GW: Direct synchronous fan-out
(acceptable for ≤256) and Notify offline members CS->>GW: Push notification to offline members end M-->>CS: Delivery confirmations CS->>A: Update delivery status
(cached in Redis) par Store for sender CS->>CASS: Store in A's partition and Store for each member loop For each member CS->>CASS: Store in member's partition
(group_id, msg_id, seq_num) end end CS->>REDIS: Get online members & gateways par Fan-out to online members CS->>GW: Forward to member B CS->>GW: Forward to member C CS->>GW: Forward to member D Note over CS,GW: Direct synchronous fan-out
(acceptable for ≤256) and Notify offline members CS->>GW: Push notification to offline members end M-->>CS: Delivery confirmations CS->>A: Update delivery status
Large Group Message Flow (257-1024 members)
| Step | Action | Rationale |
|---|---|---|
| 1. Receive | Chat service receives message, assigns group sequence number | Ensure ordering within group |
| 2. Store | Store in sender's partition + group message table | Single write, not per-member (optimization) |
| 3. Queue | Publish to Kafka topic: group_messages_{shard} | Async processing, don't block sender |
| 4. Workers | Fan-out workers consume from Kafka, batch members into chunks of 100 | Parallelized delivery |
| 5. Deliver | Workers call chat service to deliver to online members | Reuse existing delivery logic |
| 6. Lazy Load | Offline members receive pointer to message when they come online | Pull model for offline users |
Group Sequence Numbers
| Aspect | Design | Guarantees |
|---|---|---|
| Generation | Redis INCR per group_id: seq:{group_id} | Monotonic, unique per group |
| Ordering | Members apply messages in sequence order | Consistent view across all members |
| Gap Detection | Client detects missing sequence (e.g., 42, 44) → request 43 | Eventual consistency repair |
| Sync on Rejoin | Client sends last_seen_seq → server returns missed messages | Catch-up after offline period |
Interview Callout: Why Not E2E Encrypt Groups Normally?
Sender Keys protocol: Each group member has a sender key, distributed encrypted to each member. When A sends message in group, A uses their sender key to encrypt (symmetric). Receivers decrypt with A's sender key. When membership changes, sender keys rotated. Avoids N² encryption operations (encrypting once per member). WhatsApp uses Signal's Sender Keys protocol.
7. End-to-End Encryption
Signal Protocol Overview
| Component | Purpose | Cryptography |
|---|---|---|
| X3DH (Extended Triple Diffie-Hellman) | Initial key agreement when users first communicate | Curve25519 elliptic curve DH |
| Double Ratchet | Ongoing per-message key derivation and forward secrecy | DH ratchet + symmetric key ratchet (HMAC) |
| Prekey Bundle | Published signed public keys for asynchronous setup | Identity key + signed prekey + one-time prekeys |
| AES-256-CBC | Message content encryption | Symmetric encryption with derived keys |
| HMAC-SHA256 | Message authentication | Prevent tampering |
X3DH Key Exchange Flow
sequenceDiagram
participant A as Alice
participant KS as Key Server
participant B as Bob
Note over B: Registration
B->>B: Generate Identity Key IK_B
B->>B: Generate Signed Prekey SPK_B
B->>B: Generate 100 One-Time Prekeys OPK_B[1..100]
B->>KS: Upload prekey bundle
{IK_B, SPK_B, OPK_B[1..100]} Note over A: Initiate conversation A->>KS: Fetch Bob's prekey bundle KS-->>A: {IK_B, SPK_B, OPK_B[42]} KS->>KS: Mark OPK_B[42] as used A->>A: Generate ephemeral key EK_A A->>A: Perform 4 DH operations:
DH1 = DH(IK_A, SPK_B)
DH2 = DH(EK_A, IK_B)
DH3 = DH(EK_A, SPK_B)
DH4 = DH(EK_A, OPK_B[42]) A->>A: Derive shared secret SK = KDF(DH1 || DH2 || DH3 || DH4) A->>B: Initial message + {IK_A, EK_A, OPK_ID=42} B->>B: Perform same 4 DH operations B->>B: Derive same SK B->>A: Response encrypted with SK Note over A,B: Now both have shared secret SK
Transition to Double Ratchet
{IK_B, SPK_B, OPK_B[1..100]} Note over A: Initiate conversation A->>KS: Fetch Bob's prekey bundle KS-->>A: {IK_B, SPK_B, OPK_B[42]} KS->>KS: Mark OPK_B[42] as used A->>A: Generate ephemeral key EK_A A->>A: Perform 4 DH operations:
DH1 = DH(IK_A, SPK_B)
DH2 = DH(EK_A, IK_B)
DH3 = DH(EK_A, SPK_B)
DH4 = DH(EK_A, OPK_B[42]) A->>A: Derive shared secret SK = KDF(DH1 || DH2 || DH3 || DH4) A->>B: Initial message + {IK_A, EK_A, OPK_ID=42} B->>B: Perform same 4 DH operations B->>B: Derive same SK B->>A: Response encrypted with SK Note over A,B: Now both have shared secret SK
Transition to Double Ratchet
Double Ratchet Algorithm
| Ratchet Type | Mechanism | Benefit |
|---|---|---|
| Symmetric Key Ratchet | KDF chain: next_key = HMAC(current_key, constant) | Each message uses unique key, forward secrecy |
| DH Ratchet | New DH keypair with each message turn (send/receive) | Break key compromise: future keys safe even if current compromised |
| Message Keys | Derived per-message from chain key: msg_key = HMAC(chain_key, 0x01) | Unique AES key + HMAC key per message |
Properties Achieved
| Property | Description | How Achieved |
|---|---|---|
| End-to-End Confidentiality | Only sender and recipient can read message | Server never has decryption keys |
| Forward Secrecy | Past messages safe if keys compromised | Keys deleted after use, cannot decrypt past |
| Future Secrecy (Break-in Recovery) | Future messages safe if keys compromised | DH ratchet generates new shared secrets |
| Authenticity | Verify message from claimed sender | HMAC with shared secret |
| Deniability | No cryptographic proof of who sent message | Symmetric keys (both parties have same key) |
| Asynchronous | Work without both parties online | Prekey bundles stored on server |
Key Server Storage
| Key Type | Rotation | Storage (PostgreSQL) |
|---|---|---|
| Identity Key (IK) | Never (tied to account) | users.identity_key_public |
| Signed Prekey (SPK) | Weekly | signed_prekeys (keep last 2) |
| One-Time Prekeys (OPK) | As consumed (client uploads new batch when <20 remain) | one_time_prekeys (100 per user) |
Interview Callout: Server Trust Model
Server can't read messages BUT: Server can perform MITM by serving fake prekeys. Mitigation: Safety Numbers (fingerprint of identity keys). Users can out-of-band verify 60-digit code. If codes match, no MITM. WhatsApp shows security notification if recipient's identity key changes (suggests re-verification).
8. Media Handling
Media Upload Flow
sequenceDiagram
participant C as Client
participant MS as Media Service
participant S3 as S3/CDN
participant TH as Thumbnail Worker
participant CS as Chat Service
participant R as Recipient
C->>MS: 1. Request upload URL
{file_type, file_size} MS->>MS: 2. Validate (max 2GB, allowed type) MS->>MS: 3. Generate upload_id & signed URL MS-->>C: 4. {upload_id, signed_url, ttl: 5min} C->>C: 5. Encrypt media locally
(AES-256, random media_key) C->>S3: 6. Upload encrypted file to signed URL S3-->>C: 7. Upload complete C->>MS: 8. Confirm upload
{upload_id} MS->>S3: 9. Verify object exists alt Image/Video MS->>TH: 10. Generate thumbnail
(async job) TH->>S3: 11. Download encrypted file TH->>TH: 12. Decrypt with media_key
Resize to 200x200
Re-encrypt TH->>S3: 13. Upload thumbnail TH->>MS: 14. Thumbnail ready end MS-->>C: 15. {media_url, thumbnail_url} C->>CS: 16. Send message
{media_url, media_key encrypted, thumbnail_url} CS->>R: 17. Forward message R->>S3: 18. Download encrypted media R->>R: 19. Decrypt with media_key R->>R: 20. Display media
{file_type, file_size} MS->>MS: 2. Validate (max 2GB, allowed type) MS->>MS: 3. Generate upload_id & signed URL MS-->>C: 4. {upload_id, signed_url, ttl: 5min} C->>C: 5. Encrypt media locally
(AES-256, random media_key) C->>S3: 6. Upload encrypted file to signed URL S3-->>C: 7. Upload complete C->>MS: 8. Confirm upload
{upload_id} MS->>S3: 9. Verify object exists alt Image/Video MS->>TH: 10. Generate thumbnail
(async job) TH->>S3: 11. Download encrypted file TH->>TH: 12. Decrypt with media_key
Resize to 200x200
Re-encrypt TH->>S3: 13. Upload thumbnail TH->>MS: 14. Thumbnail ready end MS-->>C: 15. {media_url, thumbnail_url} C->>CS: 16. Send message
{media_url, media_key encrypted, thumbnail_url} CS->>R: 17. Forward message R->>S3: 18. Download encrypted media R->>R: 19. Decrypt with media_key R->>R: 20. Display media
Media Storage Strategy
| Aspect | Design | Rationale |
|---|---|---|
| Storage | S3 (Standard → Infrequent Access after 30 days → Glacier after 1 year) | Cost optimization, rare re-downloads of old media |
| CDN | CloudFront CDN with edge caching (TTL: 7 days) | Low latency for popular/recent media |
| Encryption | Client-side AES-256 with random key per file | E2E encryption extends to media (server can't decrypt) |
| Key Distribution | Media key sent in encrypted message payload | Only recipients can decrypt media |
| Thumbnails | Separate S3 object, also encrypted, smaller key in message | Fast preview before full download |
| Retention | 18 months (then deleted unless backed up by user) | Balance storage costs vs. user expectations |
Media Types & Processing
| Type | Max Size | Processing | Thumbnail |
|---|---|---|---|
| Image (JPEG/PNG/WebP) | 16MB | Compress to max 1920x1080, strip EXIF | 200x200 JPEG |
| Video (MP4/MOV) | 2GB | Transcode to H.264/AAC, max 1280x720 | First frame 200x200 |
| Audio (MP3/AAC/Opus) | 100MB | Transcode to Opus 32kbps | Waveform image |
| Document (PDF/DOCX/etc) | 2GB | Virus scan, no transcoding | First page render (PDF only) |
| Voice Note | 10MB | Opus 16kbps | Waveform |
Upload Resume & Chunking
| Scenario | Solution |
|---|---|
| Large files (>100MB) | S3 Multipart Upload: split into 10MB chunks, upload in parallel |
| Network interruption mid-upload | Client tracks uploaded chunks, resumes from last successful chunk |
| Duplicate uploads | Content hash (SHA-256) deduplication: if hash exists, return existing URL |
| Slow upload on mobile | Progressive upload: send thumbnail + low-res first, full-res in background |
Interview Callout: Auto-Download Settings
Bandwidth management: Client-side settings control auto-download: WiFi (all media), Cellular (photos only <5MB), Never. Server sends message with media metadata (size, type, thumbnail). Client decides whether to auto-download. Reduces bandwidth costs, especially important in emerging markets with expensive/slow mobile data.
9. Delivery Guarantees
Message Status States
| State | Icon | Meaning | Trigger |
|---|---|---|---|
| Pending | 🕐 (clock) | Client sending to server | Message in outbox, not yet ACKed by server |
| Sent | ✓ (single gray tick) | Received by server, persisted | Server ACK to sender |
| Delivered | ✓✓ (double gray ticks) | Delivered to recipient's device | Recipient client ACK to server |
| Read | ✓✓ (double blue ticks) | Viewed by recipient | User opened chat, message in viewport |
| Failed | ⚠️ (warning) | Delivery failed after retries | User blocked, number invalid, or server error |
Exactly-Once Delivery Mechanism
| Component | Implementation | Guarantees |
|---|---|---|
| Message ID | Client generates UUID before send | Idempotency key for deduplication |
| Dedup Cache | Redis bloom filter (7-day TTL) | Fast lookup: O(1), low false positive rate (0.01%) |
| Retry Logic | Exponential backoff: 1s, 2s, 4s, 8s, 16s, then fail | Handle transient failures without flooding |
| ACK Protocol | Server ACKs with message_id, client matches | Client knows exactly which message succeeded |
| Persistence First | Write to Cassandra BEFORE ACK to sender | Never lose message after ACK |
Message Ordering
| Scope | Guarantee | Implementation |
|---|---|---|
| Per-Chat (1-to-1) | Strict FIFO from each sender | Client-side sequence number + server timestamp |
| Group Chat | Total order across all senders | Server assigns group-wide sequence number (Redis INCR) |
| Multi-Device (same user) | Eventually consistent, no strict order | Lamport timestamps for conflict resolution |
| Out-of-Order Arrival | Client detects gaps, requests missing messages | Sequence number gap detection: if recv 42 then 44, request 43 |
Offline Message Queue
| Aspect | Design |
|---|---|
| Storage | Redis sorted set per user: offline_queue:{user_id} (score = timestamp) |
| Retention | 30 days (after that, still in Cassandra but not pushed) |
| Batch Delivery | When user comes online, fetch all queued messages in single request |
| Pagination | If >1000 messages, deliver in batches of 1000 (avoid overwhelming client) |
| Priority | Text messages first, then media pointers (media downloaded on-demand) |
| Push Notification | First message → push notification, subsequent messages batched |
Read Receipts
sequenceDiagram
participant S as Sender
participant CS as Chat Service
participant R as Recipient
S->>CS: Send message {msg_id: 123}
CS->>S: ACK (sent ✓)
CS->>R: Deliver message
R->>CS: ACK received {msg_id: 123}
CS->>S: Update status (delivered ✓✓)
Note over R: User opens chat,
message in viewport R->>CS: Read receipt {msg_id: 123, read_at: timestamp} CS->>CS: Update message status in DB CS->>S: Update status (read ✓✓ blue) Note over S: Sender sees blue ticks
message in viewport R->>CS: Read receipt {msg_id: 123, read_at: timestamp} CS->>CS: Update message status in DB CS->>S: Update status (read ✓✓ blue) Note over S: Sender sees blue ticks
Interview Callout: Why Two Storage Writes?
Sender + Recipient partitions: Cassandra optimized for writes, not cross-partition queries. Storing message in both sender's and recipient's partition allows fast queries ("show my chat with Bob") without scatter-gather. Cost: 2× storage. Benefit: O(1) reads, no hot partitions. Alternative (worse): single storage with secondary index on (sender, recipient) → slower, potential hotspots.
10. Scalability & Architecture Decisions
Erlang/BEAM for WebSocket Gateways
| Feature | Capability | WhatsApp's Results |
|---|---|---|
| Lightweight Processes | 2KB per process (vs. 2MB thread in Java) | 2M+ concurrent connections per server (32GB RAM) |
| Preemptive Scheduling | Each process gets fair CPU time (2000 reductions) | No single slow connection blocks others |
| Fault Isolation | Process crash doesn't affect others | One bad connection → kill process, others unaffected |
| Hot Code Reloading | Upgrade code without dropping connections | Zero-downtime deploys |
| Distribution | Built-in clustering and message passing | Seamless scale-out across servers |
Sharding Strategy
| Component | Shard Key | Sharding Method |
|---|---|---|
| WebSocket Gateway | user_id % num_gateways | Consistent hashing (session stickiness) |
| Cassandra (messages) | user_id (partition key) | Cassandra's native partitioning |
| PostgreSQL (users/groups) | user_id or group_id | Hash-based sharding (Vitess or Citus) |
| Redis (presence) | user_id % num_clusters | Redis Cluster (hash slots) |
| Kafka (message stream) | user_id or group_id | Kafka partition by key hash |
Multi-Region Architecture
| Aspect | Design | Trade-offs |
|---|---|---|
| User Affinity | User pinned to home region (nearest at registration) | Low latency for most traffic, complexity for migrations |
| Cross-Region Messaging | Gateway in recipient's region fetches from sender's region Cassandra | Higher latency for cross-region (100-200ms), but rare (most chats regional) |
| Media Replication | S3 cross-region replication for media | Faster downloads globally, 2× storage cost |
| Disaster Recovery | Cassandra multi-DC replication (async), can failover to DR region | RPO ~5 seconds (async lag), RTO ~10 minutes (DNS switch) |
| Global Metadata | User profiles replicated globally (PostgreSQL streaming replication) | Eventual consistency (seconds), worth it for low latency reads |
Capacity Planning (Per Region)
| Component | Instance Type | Count | Cost Estimate |
|---|---|---|---|
| WebSocket Gateways | c6g.16xlarge (64 vCPU, 128GB) | 25 | $55K/month |
| Chat Service | c6g.8xlarge (32 vCPU, 64GB) | 70 | $77K/month |
| Cassandra Cluster | i4i.8xlarge (32 vCPU, 256GB, 7.5TB NVMe) | 350 | $1.4M/month |
| Redis (Presence) | r6g.4xlarge (16 vCPU, 128GB) | 500 | $770K/month |
| Kafka Brokers | m6g.2xlarge (8 vCPU, 32GB) | 30 | $14K/month |
| S3 Storage (media) | S3 Standard → IA → Glacier | 5.5EB total | ~$60M/month (blended rate) |
| Total/Region | ~$62M/month | ||
| Total (5 regions) | ~$310M/month |
Performance Optimizations
| Optimization | Impact |
|---|---|
| Protocol Buffers over WebSocket | 50% smaller payload vs. JSON → less bandwidth |
| Message Batching | Send up to 50 messages in single WebSocket frame → fewer syscalls |
| Lazy Load Media | Send thumbnail first, full media only when user clicks → faster perception |
| Bloom Filter Deduplication | O(1) dedup check vs. DB query → 100× faster |
| Cassandra Write-Through Cache | Recent 100 messages per chat in Redis → 80% cache hit rate |
| Connection Draining | Graceful shutdown: gateway stops accepting new, drains existing over 60s → zero dropped messages |
| gRPC Connection Pooling | Reuse connections between gateway ↔ chat service → less TLS overhead |
Interview Callouts Summary
| Topic | Key Insight |
|---|---|
| Erlang vs. Go for Gateways | Erlang's lightweight processes enable 2M connections/server; Go maxes ~1M |
| Dual Message Storage | 2× storage cost for O(1) read performance, avoids hot partitions |
| Presence Fan-Out Optimization | Only broadcast to active chats, not all contacts → 100× less traffic |
| Group Size Threshold (256) | Sync fan-out ≤256 members, async (Kafka) >256 → balance latency & load |
| E2E Encryption Server Trust | Server can MITM prekeys; safety numbers provide out-of-band verification |
| Media Encryption | Client-side encryption extends E2E to media; key in message, server can't decrypt |
| Exactly-Once Delivery | Client-generated UUID + bloom filter deduplication + persist-before-ACK |
| Multi-Region Latency | User affinity to home region; cross-region adds 100-200ms but rare |
Operational Readiness (Production-Grade)
Service Level Indicators (SLIs)
- Message delivery P99 < 500ms: Time from sender submit to recipient device ACK (excludes offline queueing)
- Connection success > 99.9%: Ratio of successful WebSocket connections to total connection attempts
- E2E encryption handshake < 2s: X3DH key exchange time for first message to new contact
Monitoring & Alerting
- WebSocket connection count: Track active connections per gateway (alert if > 1.8M per server, scale out)
- Message queue depth: Monitor Kafka lag for async processing (alert if > 100K messages, indicates worker saturation)
- Delivery receipt latency: Measure time from message sent to delivered status update (alert if P95 > 2s)
- Presence update fan-out rate: Track presence broadcast volume (alert if > 500K updates/sec, indicates broadcast storm)
Deployment Strategy
- Rolling restart of gateway servers: Drain connections first (send reconnect signal to clients), wait 60s for graceful disconnect, ensure 0 dropped messages during deploy
Circuit Breakers
- Chat service → presence service: If presence service latency > 500ms or error rate > 10%, degrade by hiding online status (messages still work)
- Media → thumbnail generator: If thumbnail generation fails, send placeholder thumbnail, generate async and push update later
Disaster Recovery
- Messages stored in recipient's region: Cassandra multi-DC replication within region, no cross-region replication (data sovereignty)
- Multi-region gateway: If one region fails, users reconnect to nearest available gateway (GeoDNS failover), RPO: 0 (synchronous writes within region)