Chapter 48: Design Twitter/X

Case Study: Design a social media platform like Twitter/X that allows users to post short messages (tweets), follow other users, and view timelines of tweets from people they follow. The system must handle hundreds of millions of users, support real-time trending topics, and provide fast timeline generation.

Key Challenge: The fanout problem - when a celebrity with 50 million followers posts a tweet, how do you efficiently deliver it to all their followers' timelines without overwhelming the system?

1. Requirements

Functional Requirements

Non-Functional Requirements

Interview Tip: Clarify the read-to-write ratio early. Twitter is extremely read-heavy (1000:1 ratio). This fundamentally shapes the architecture - you can afford to do expensive work at write time to make reads fast. Also confirm that eventual consistency is acceptable for timelines (it is).

Out of Scope

2. Capacity Estimation

Traffic Estimates

Daily Active Users: 300M DAU

Tweets per day: 300M users × 2 tweets/day = 600M tweets/day

Tweets per second (average): 600M / 86,400 = 7,000 tweets/sec

Tweets per second (peak): 7,000 × 3 = 21,000 tweets/sec

Timeline reads per day: 300M users × 20 timeline views/day = 6B timeline reads/day

Timeline reads per second (average): 6B / 86,400 = 70,000 reads/sec

Timeline reads per second (peak): 70,000 × 3 = 210,000 reads/sec

Read-to-write ratio: 70,000 / 7,000 = 10:1 (actually much higher in practice, closer to 100:1)

Storage Estimates

Tweet size:

  • Tweet text: 280 chars × 2 bytes (UTF-8) = 560 bytes
  • Metadata (user_id, timestamp, tweet_id, etc.): 200 bytes
  • Total per tweet: ~800 bytes ≈ 1 KB

Daily storage: 600M tweets × 1 KB = 600 GB/day

5-year storage: 600 GB × 365 × 5 = 1.1 PB

Media storage:

  • 20% of tweets have images (avg 200 KB each)
  • 5% of tweets have videos (avg 2 MB each)
  • Daily media: (600M × 0.2 × 200KB) + (600M × 0.05 × 2MB) = 24TB + 60TB = 84TB/day
  • 5-year media storage: 84TB × 365 × 5 = 153 PB

Fanout Calculation (Critical)

Average user has 200 followers

Fanout work per tweet: 1 tweet × 200 followers = 200 timeline writes

Total fanout writes/sec: 7,000 tweets/sec × 200 followers = 1.4M timeline writes/sec

Celebrity problem: User with 50M followers posts → 50M timeline writes for a single tweet

At 10,000 writes/sec per server, this single tweet requires 5,000 servers for several seconds - NOT FEASIBLE

Interview Tip: The fanout calculation is critical. Show the math that breaks pure fanout-on-write for celebrities. This naturally leads to the hybrid approach. Many candidates miss this and propose naive solutions that don't scale.

Cache Estimates

Timeline cache: Store 1,000 recent tweets per user's timeline

  • 300M users × 1,000 tweets × 8 bytes (tweet_id) = 2.4 TB
  • Use Redis sorted sets (score = timestamp)
  • Only cache active users (80/20 rule): 2.4TB × 0.2 = 480 GB

3. High-Level Architecture

graph TB Client[Client App/Web] subgraph "API Gateway" LB[Load Balancer] API[API Servers] end subgraph "Core Services" TS[Tweet Service] TLS[Timeline Service] US[User Service] SS[Search Service] NS[Notification Service] TS2[Trending Service] end subgraph "Fanout System" FQ[Fanout Queue
Kafka] FW[Fanout Workers
Worker Pool] end subgraph "Storage" TDB[(Tweet DB
Cassandra)] UDB[(User/Graph DB
PostgreSQL)] TC[Timeline Cache
Redis] MC[Media Storage
S3/CDN] end subgraph "Search & Analytics" ES[Elasticsearch
Inverted Index] TQ[Trending Queue
Kafka] TAgg[Trending Aggregator
Stream Processing] end Client -->|Posts/Reads| LB LB --> API API --> TS API --> TLS API --> US API --> SS API --> NS TS -->|Write tweet| TDB TS -->|Upload media| MC TS -->|Publish| FQ TS -->|Index| ES TS -->|Track| TQ FQ --> FW FW -->|Write timelines| TC FW -->|Notify| NS TLS -->|Read cache| TC TLS -->|Read tweets| TDB TLS -->|Merge celebrity| TS US --> UDB SS --> ES TQ --> TAgg TAgg --> TS2

Component Responsibilities

Interview Tip: Draw this architecture on the whiteboard first, then deep-dive into each component. The interviewer will likely focus on fanout and timeline generation - be ready to explain both in detail with sequence diagrams.

4. Tweet Ingestion

Write Path

sequenceDiagram participant Client participant API participant TweetService participant TweetDB participant MediaService participant FanoutQueue participant SearchIndex participant TrendingQueue Client->>API: POST /tweets {text, media} API->>TweetService: createTweet(userId, text, media) alt Has media TweetService->>MediaService: uploadMedia(media) MediaService-->>TweetService: mediaUrl end TweetService->>TweetService: validate(text, media) TweetService->>TweetService: shortenUrls(text) TweetService->>TweetService: generateTweetId() TweetService->>TweetDB: INSERT tweet TweetDB-->>TweetService: success par Async operations TweetService->>FanoutQueue: publish(tweetId, userId) TweetService->>SearchIndex: index(tweetId, text) TweetService->>TrendingQueue: extractHashtags(text) end TweetService-->>API: tweetId API-->>Client: 201 Created {tweetId}

Tweet Validation

Media Upload

URL Shortening

Twitter shortens all URLs to save character count and track clicks:

  • User posts: "Check out https://www.example.com/very/long/article"
  • System shortens to: "Check out https://t.co/abc123"
  • Shortener service maps abc123 → original URL
  • Track click analytics (clicks, referrers, geographic data)

Tweet ID Generation

Snowflake ID: Twitter's custom 64-bit ID format

  • 41 bits: Timestamp (milliseconds since epoch) - sortable by time
  • 10 bits: Machine ID (1024 machines)
  • 12 bits: Sequence number (4096 IDs per millisecond per machine)
  • Total: 4M IDs per second across all machines
  • Benefits: No coordination needed, sortable, unique

5. Fanout Strategies

The Fanout Problem

When a user posts a tweet, that tweet must appear in the timelines of all their followers. This process of "fanning out" a single tweet to many timelines is the core challenge of Twitter's architecture.

Option 1: Fanout-on-Write (Push Model)

Strategy: When a user posts, immediately write that tweet to all followers' timeline caches.

graph LR User[User Posts Tweet] Q[Fanout Queue] W1[Worker 1] W2[Worker 2] W3[Worker N] T1[Follower 1
Timeline Cache] T2[Follower 2
Timeline Cache] T3[Follower N
Timeline Cache] User -->|Publish| Q Q --> W1 Q --> W2 Q --> W3 W1 -->|Write tweetId| T1 W2 -->|Write tweetId| T2 W3 -->|Write tweetId| T3

Fanout-on-Write Process

  1. User posts tweet → Tweet Service writes to database
  2. Tweet Service publishes message to Kafka: {tweetId, userId}
  3. Fanout workers consume message and query User Service for follower list
  4. For each follower, worker writes tweetId to that follower's Redis timeline cache
  5. Timeline is a Redis sorted set: key=user:{userId}:timeline, score=timestamp

Pros

Cons

Math: Why pure fanout-on-write fails

Celebrity with 50M followers posts 1 tweet:

  • Need to write to 50M timeline caches
  • Redis write: ~1ms per write (including network)
  • Single-threaded: 50M × 1ms = 50,000 seconds = 13.9 hours
  • With 1,000 parallel workers: 50M / 1,000 = 50,000 writes per worker = 50 seconds
  • If celebrity posts 10 tweets/day, system is constantly backlogged

Option 2: Fanout-on-Read (Pull Model)

Strategy: Don't pre-compute timelines. When a user requests their timeline, fetch recent tweets from all the people they follow and merge them.

graph LR User["User Requests Timeline"] TS[Timeline Service] US[User Service] TDB[(Tweet DB)] User --> TS TS -->|Get following list| US US -->|user1, user2, ..., userN| TS TDB -->|tweets| TS TS -->|Merge & sort| TS TS -->|timeline| User

Fanout-on-Read Process

  1. User requests timeline → Timeline Service queries User Service for following list
  2. Timeline Service queries Tweet DB: "Get 100 recent tweets from user1, user2, ..., userN"
  3. Merge all tweets and sort by timestamp (merge K sorted lists)
  4. Return top 100 tweets

Pros

Cons

Interview Tip: Explain that fanout-on-read is theoretically simple but fails at Twitter's scale. 600K timeline reads/sec, each querying 200 users, creates 120M database queries/sec - not feasible. This is why the hybrid approach is necessary.

Option 3: Hybrid Approach (Twitter's Solution)

Strategy: Use fanout-on-write for regular users, fanout-on-read for celebrities. Merge both at read time.

graph TB User[User Posts Tweet] Check{Celebrity?
Followers > 1M} subgraph "Regular User Path" FOW[Fanout on Write] Queue[Fanout Queue] Workers[Worker Pool] Cache[Timeline Caches] FOW --> Queue Queue --> Workers Workers --> Cache end subgraph "Celebrity Path" NoFanout[Skip Fanout] CelebDB[(Celebrity
Tweet DB)] NoFanout --> CelebDB end User --> Check Check -->|No| FOW Check -->|Yes| NoFanout subgraph "Timeline Read" Request[User Requests
Timeline] TService[Timeline Service] ReadCache[Read from Cache
Fanout Tweets] ReadCeleb[Read Celebrity
Tweets] Merge[Merge & Sort] Request --> TService TService --> ReadCache TService --> ReadCeleb ReadCache --> Merge ReadCeleb --> Merge end Cache -.-> ReadCache CelebDB -.-> ReadCeleb

Hybrid Algorithm

Write Path: if (user.followers < 1,000,000): // Regular user - fanout on write publish_to_fanout_queue(tweet_id, user_id) else: // Celebrity - skip fanout mark_as_celebrity_tweet(tweet_id, user_id) Read Path: function getTimeline(userId): // Get fanout tweets from cache (pre-computed) cachedTweets = redis.zrevrange("user:#{userId}:timeline", 0, 99) // Get list of celebrities this user follows celebrities = db.query("SELECT following_id FROM follows WHERE follower_id = ? AND following_id IN (celebrity_users)", userId) // Fetch recent tweets from celebrities celebrityTweets = db.query("SELECT * FROM tweets WHERE user_id IN (?) ORDER BY timestamp DESC LIMIT 100", celebrities) // Merge both lists and sort by timestamp allTweets = merge_sort(cachedTweets, celebrityTweets, by=timestamp) return allTweets[0:100]

Pros

Cons

Threshold Tuning

The "celebrity" threshold can be tuned based on system performance:

  • Conservative (100K followers): More users in celebrity bucket, less write amplification, slightly slower reads
  • Aggressive (10M followers): Fewer celebrities, more write amplification, faster reads for most users
  • Twitter's approach: Around 1M followers threshold
  • Dynamic adjustment: Can adjust threshold during peak events (e.g., elections, sports events)

6. Home Timeline Generation

Timeline Cache Structure

Each user's timeline is stored in Redis as a sorted set:

  • Key: "user:{userId}:timeline"
  • Score: Tweet timestamp (for chronological ordering)
  • Value: Tweet ID (lightweight, actual tweet data fetched separately)
  • Size: Store only 1,000 most recent tweets per user (5-10 days of content)
Redis Sorted Set: Key: "user:12345:timeline" ZADD user:12345:timeline 1652345678000 "tweet_id_1" ZADD user:12345:timeline 1652345679000 "tweet_id_2" ZADD user:12345:timeline 1652345680000 "tweet_id_3" // Retrieve 100 most recent tweets ZREVRANGE user:12345:timeline 0 99 // Trim to keep only 1000 most recent ZREMRANGEBYRANK user:12345:timeline 0 -1001

Timeline Read Sequence

sequenceDiagram participant Client participant API participant TimelineService participant RedisCache participant UserService participant TweetDB Client->>API: GET /timeline API->>TimelineService: getTimeline(userId) par Fetch cached tweets TimelineService->>RedisCache: ZREVRANGE user:{id}:timeline 0 99 RedisCache-->>TimelineService: [tweet_ids] and Fetch celebrity list TimelineService->>UserService: getCelebrityFollowing(userId) UserService-->>TimelineService: [celebrity_ids] end alt User follows celebrities TimelineService->>TweetDB: SELECT * FROM tweets WHERE user_id IN (celebrity_ids) LIMIT 100 TweetDB-->>TimelineService: [celebrity_tweets] TimelineService->>TimelineService: merge(cached_tweets, celebrity_tweets) end TimelineService->>TweetDB: batchGetTweets([tweet_ids]) TweetDB-->>TimelineService: [tweet_objects] TimelineService-->>API: timeline API-->>Client: 200 OK [tweets]

Timeline Ranking (Optional)

Modern Twitter uses a ranked timeline instead of purely chronological:

  • Engagement signals: Likes, retweets, replies count
  • User affinity: How often you interact with the tweet author
  • Recency: Recent tweets score higher
  • Media: Tweets with images/videos score higher
  • ML model: Predict probability user will engage with tweet

For simplicity, this design focuses on chronological timeline, but a ranking layer can be added on top.

Timeline Pagination

Cursor-based pagination: GET /timeline?cursor=null // First page → Returns tweets 1-100 and cursor="tweet_100_timestamp" GET /timeline?cursor=tweet_100_timestamp // Next page → Returns tweets 101-200 and cursor="tweet_200_timestamp" Redis implementation: ZREVRANGEBYSCORE user:{id}:timeline {cursor} -inf LIMIT 0 100

Cache Invalidation

9. Database Design

Tweets Table

CREATE TABLE tweets ( tweet_id BIGINT PRIMARY KEY, -- Snowflake ID user_id BIGINT NOT NULL, -- Author text VARCHAR(280) NOT NULL, -- Tweet content media_urls TEXT[], -- Array of media URLs created_at TIMESTAMP NOT NULL, -- Tweet timestamp reply_to_tweet_id BIGINT, -- NULL if not a reply retweet_of_tweet_id BIGINT, -- NULL if not a retweet like_count INT DEFAULT 0, retweet_count INT DEFAULT 0, reply_count INT DEFAULT 0, INDEX idx_user_created (user_id, created_at DESC) );

Users Table

CREATE TABLE users ( user_id BIGINT PRIMARY KEY, username VARCHAR(50) UNIQUE NOT NULL, display_name VARCHAR(100), bio VARCHAR(500), profile_image_url VARCHAR(500), follower_count INT DEFAULT 0, following_count INT DEFAULT 0, created_at TIMESTAMP NOT NULL, is_celebrity BOOLEAN DEFAULT FALSE -- For hybrid fanout );

Follows Table (Social Graph)

CREATE TABLE follows ( follower_id BIGINT NOT NULL, -- User who follows following_id BIGINT NOT NULL, -- User being followed created_at TIMESTAMP NOT NULL, -- When follow happened PRIMARY KEY (follower_id, following_id), INDEX idx_following (following_id, follower_id) -- For fanout );

Sharding Strategy

Option 1: Shard by tweet_id

  • Sharding key: tweet_id % num_shards
  • Pros: Even distribution, simple
  • Cons: User timeline query requires scatter-gather across all shards

Option 2: Shard by user_id

  • Sharding key: user_id % num_shards
  • Pros: User timeline query hits single shard
  • Cons: Uneven distribution (celebrities generate far more tweets)
  • Cons: Hotspot issue (celebrity tweets hit one shard repeatedly)

Recommended: Hybrid Sharding

Approach: Use Cassandra with composite partition key

  • Partition key: (user_id, tweet_id / 1000000) - groups tweets by user and time bucket
  • Clustering key: tweet_id (sorted within partition)
  • Query: SELECT * FROM tweets WHERE user_id = ? ORDER BY tweet_id DESC LIMIT 100
  • Result: Single partition read, time-ordered results

Cassandra Schema

CREATE TABLE tweets ( user_id BIGINT, time_bucket BIGINT, -- tweet_id / 1000000 (groups ~1M tweets) tweet_id BIGINT, text VARCHAR, media_urls LIST<TEXT>, created_at TIMESTAMP, reply_to_tweet_id BIGINT, retweet_of_tweet_id BIGINT, like_count INT, retweet_count INT, reply_count INT, PRIMARY KEY ((user_id, time_bucket), tweet_id) ) WITH CLUSTERING ORDER BY (tweet_id DESC); Query examples: -- Get user's recent tweets (single partition read) SELECT * FROM tweets WHERE user_id = 123 AND time_bucket = 1652 LIMIT 100; -- Get specific tweet (need to know user_id and time_bucket, or use secondary index) SELECT * FROM tweets WHERE tweet_id = 123456789;

Timeline Cache Schema (Redis)

Sorted set for timeline: Key: "user:{user_id}:timeline" Score: tweet timestamp Value: tweet_id Commands: ZADD user:123:timeline 1652345678000 tweet_1 ZREVRANGE user:123:timeline 0 99 -- Get 100 most recent ZREMRANGEBYRANK user:123:timeline 0 -1001 -- Keep only 1000 recent

Database Selection

10. Notification System

Notification Types

Notification Pipeline

graph LR Event[Tweet/Like/Follow
Event] Kafka[Kafka:
notification-events] subgraph "Notification Service" Router[Event Router] Aggregator[Notification
Aggregator] Filter[User
Preferences
Filter] Prioritizer[Priority
Queue] end Push[Push
Notification
Service] Email[Email
Service] DB[(Notification DB)] Event --> Kafka Kafka --> Router Router --> Aggregator Aggregator --> Filter Filter --> Prioritizer Prioritizer --> Push Prioritizer --> Email Prioritizer --> DB

Notification Aggregation

Instead of sending 100 notifications for 100 likes, aggregate them:

  • Batching window: Wait 5 minutes before sending notification
  • Aggregation: "User X and 99 others liked your tweet"
  • Reduces spam: Popular tweets get thousands of likes - don't send thousands of notifications

Priority Queue

Not all notifications are equally important:

  • High priority: Direct messages, mentions from verified users
  • Medium priority: Likes, retweets, replies
  • Low priority: New followers, trending topics
  • Implementation: Separate Kafka partitions or SQS queues per priority

User Preferences

Allow users to control notification settings:

  • User can mute specific users or keywords
  • User can disable specific notification types (e.g., only mentions, no likes)
  • User can set "quiet hours" (no notifications 10pm-8am)
  • Store preferences in User DB, check before sending

Delivery Channels

Notification Storage

CREATE TABLE notifications ( notification_id BIGINT PRIMARY KEY, user_id BIGINT NOT NULL, -- Recipient type VARCHAR(50) NOT NULL, -- mention, like, retweet, etc. actor_id BIGINT, -- Who triggered the notification tweet_id BIGINT, -- Related tweet created_at TIMESTAMP NOT NULL, read_at TIMESTAMP, -- NULL if unread INDEX idx_user_created (user_id, created_at DESC) );

11. Scalability

Timeline Cache Scaling

Tweet Storage Scaling

Media Storage Scaling

Fanout Queue Scaling

Database Read Replicas

Auto-Scaling

Scale different components independently based on load:

  • API servers: Scale based on request rate (target: 1000 req/sec per server)
  • Fanout workers: Scale based on Kafka queue depth (target: under 1 minute lag)
  • Search indexers: Scale based on Kafka queue depth
  • Redis cache: Pre-provisioned (doesn't scale down, only up)

Global Distribution

graph TB subgraph "US-East Region" USE_LB[Load Balancer] USE_API[API Servers] USE_Cache[Redis Cache] USE_DB[Cassandra] end subgraph "EU-West Region" EUW_LB[Load Balancer] EUW_API[API Servers] EUW_Cache[Redis Cache] EUW_DB[Cassandra] end subgraph "Asia-Pacific Region" AP_LB[Load Balancer] AP_API[API Servers] AP_Cache[Redis Cache] AP_DB[Cassandra] end CDN[Global CDN
CloudFront] GeoRouter[GeoDNS
Route 53] User[User] --> GeoRouter GeoRouter -->|US traffic| USE_LB GeoRouter -->|EU traffic| EUW_LB GeoRouter -->|Asia traffic| AP_LB USE_API --> CDN EUW_API --> CDN AP_API --> CDN

12. Tradeoffs

Fanout-on-Write vs Fanout-on-Read

Aspect Fanout-on-Write Fanout-on-Read Hybrid
Read Latency Very fast (10ms) Slow (200-500ms) Fast (50ms)
Write Cost High (1 write → 200 writes) Low (1 write → 1 write) Medium
Celebrity Handling Fails (50M writes per tweet) Works Works
Complexity Simple Simple Complex
Consistency Eventual (fanout takes time) Consistent Eventual
Inactive Users Wasted work (still fanout) No waste (on-demand) Some waste

Write Amplification

Problem: Fanout-on-write amplifies every tweet write by the follower count.

  • User posts 1 tweet → System writes to 200 timeline caches
  • 1 tweet write becomes 200 cache writes (200x amplification)
  • 7,000 tweets/sec × 200 = 1.4M cache writes/sec
  • Mitigation: Hybrid approach skips fanout for celebrities

Read Amplification

Problem: Fanout-on-read amplifies every timeline read by the following count.

  • User requests timeline → System queries tweets from 200 users
  • 1 timeline read becomes 200 tweet queries (200x amplification)
  • 70,000 timeline reads/sec × 200 = 14M tweet queries/sec
  • Mitigation: Fanout-on-write pre-computes timelines

Consistency vs Availability

Twitter chooses availability over consistency (AP in CAP theorem):

  • Timeline staleness: Timelines may be 1-2 seconds behind (acceptable)
  • Follower count lag: Follower count may be incorrect for minutes (acceptable)
  • Cross-region lag: Tweet posted in US may take 1-2 seconds to appear in EU
  • Critical data: Tweets themselves are durable (Cassandra replication factor 3)

Storage Cost vs Latency

Timeline cache: Store tweet IDs (8 bytes) instead of full tweets (1 KB)

  • Approach 1: Cache full tweets (1 KB × 1,000 tweets × 300M users = 300 TB)
  • Approach 2: Cache tweet IDs (8 bytes × 1,000 tweets × 300M users = 2.4 TB)
  • Tradeoff: Approach 2 saves 99% memory but requires extra DB lookup (adds 10ms)
  • Decision: Use Approach 2, batch lookup tweet details (acceptable latency)

Ranking vs Simplicity

Chronological timeline: Simple, predictable, but may miss important tweets

Ranked timeline: Shows most relevant tweets first, but complex ML model

  • Twitter's evolution: Started chronological, moved to ranked (controversial)
  • User preference: Offer both options (switch in settings)
  • Implementation: Chronological is baseline, add ranking layer on top

Real-Time vs Cost

How quickly must a tweet appear in followers' timelines?

  • Real-time (1 second): Requires large worker pool, expensive
  • Near-real-time (10 seconds): Smaller worker pool, cheaper, still feels real-time
  • Delayed (1 minute): Very cheap, but poor user experience
  • Decision: Target 10-second fanout (99% of tweets), accept occasional 1-minute delays during peak
Interview Tip: Always discuss tradeoffs explicitly. Say "We're choosing X over Y because [reason], but this means [downside]." Show you understand both sides. For Twitter, the key tradeoff is write amplification (fanout) for fast reads - defend this choice with the read-to-write ratio (100:1).

Summary

Designing Twitter requires handling extreme scale (300M DAU, 600K reads/sec) while maintaining low latency for timeline generation. The core challenge is the fanout problem - efficiently delivering tweets to millions of followers.

Key Design Decisions

Capacity Estimates

Technologies

Final Interview Tips:
  1. Start with requirements and capacity estimation - this drives the architecture
  2. Draw the high-level architecture diagram first, then deep-dive into fanout and timeline
  3. Explain the celebrity problem with math - this shows you understand the constraints
  4. Discuss tradeoffs explicitly: write amplification vs read amplification, consistency vs availability
  5. Be ready to discuss edge cases: what happens when a celebrity unfollows someone? How do you handle deleted tweets?
  6. Twitter is a classic system design question - interviewers will push hard on fanout, so master this concept

Operational Readiness (Production-Grade)

A complete system design includes operational considerations. Here's what makes this Twitter design production-ready:

SLIs/SLOs (Service Level Indicators/Objectives)

Monitoring

Prometheus + Grafana dashboards track:
  • Timeline cache hit rate: Target 95%+. If drops below 90%, investigate (cache evictions, expired keys, cluster issues)
  • Fanout worker lag: Time between tweet posted and fanout complete. Target P99 < 5s. Alert if lag > 30s (indicates worker pool undersized or queue backlog)
  • Tweet ingestion QPS: Track write rate to Cassandra. Sudden drop may indicate upstream service failure
  • Trending algorithm freshness: Time since last trending update. Alert if > 15 minutes (stream processing job may be stuck)
  • Celebrity tweet merge latency: Track time spent merging celebrity tweets into timelines. Alert if P99 > 50ms (DB query too slow)
  • Kafka consumer lag: Monitor fanout queue, search indexing queue, trending queue. Alert if lag > 1 minute

Alerting

Alerting best practice: Alert on user-facing symptoms (timeline slow, post fails), not internal causes (CPU high, disk full). Symptoms directly impact users. Causes are diagnostic tools, not paging triggers.

Deployment

Feature-flagged rollouts:
  • Changes to fanout algorithm deployed behind feature flag (LaunchDarkly, ConfigCat)
  • Stage 1: Enable for 1% of users (internal employees first)
  • Stage 2: Monitor key metrics: fanout latency, timeline load time, user engagement
  • Stage 3: If metrics healthy, expand to 10% → 25% → 50% → 100% over 2 days
  • Rollback: Instant rollback via feature flag toggle (no code deploy needed)
Canary deployment for services:
  • Deploy new Timeline Service version to 5% of servers
  • Monitor error rate, latency, throughput for 15 minutes
  • If healthy, expand to 25% → 100%
  • Automated rollback if error rate > 2x baseline

Circuit Breakers

Disaster Recovery

Multi-Region Strategy: Timeline Cache (Redis): - Cache is ephemeral (can rebuild from tweet DB) - No cross-region replication (too expensive for cache) - RPO: N/A (cache is not source of truth) - RTO: 30 minutes (time to rebuild cache by replaying recent tweets) Tweet Store (Cassandra): - Multi-region replication: us-east-1, us-west-2, eu-west-1 - Replication factor: 3 per region (9 copies total globally) - Async cross-region replication (eventual consistency) - RPO: 1 minute (cross-region replication lag) - RTO: 5 minutes (DNS failover to healthy region) User/Graph DB (PostgreSQL): - Primary: us-east-1, Standby: us-west-2 - Streaming replication (<5 second lag) - RPO: 5 seconds (small data loss possible) - RTO: 10 minutes (promote standby, update DNS) DR Testing: - Quarterly game day: simulate us-east-1 outage - Verify: timeline loads work, tweet posts work, fanout continues - Measure: actual RTO achieved, gaps in runbooks

Testing

Operational Runbooks

Document common failure scenarios and remediation steps:

  • Fanout lag spike: Check Kafka consumer lag → scale workers → check for celebrity tweet storm → enable rate limiting if needed
  • Timeline cache miss rate high: Check Redis cluster health → check for memory pressure → check for eviction spikes → scale Redis cluster if needed
  • Cassandra hotspot: Identify hot partition (celebrity user) → add secondary index → consider sharding celebrity tweets separately
  • Search down: Check Elasticsearch cluster health → check for indexing backlog → restart stuck indexer workers → enable search result caching
Interview Tip: Mentioning operational readiness shows production maturity. Most candidates stop at architecture diagrams. Discussing SLOs ("P99 timeline load < 200ms") shows you think about user experience. Mentioning chaos testing ("kill random fanout workers weekly") shows you understand reliability engineering. This differentiates senior candidates from mid-level.