Chapter 53

Design Slack/Discord

1. Requirements 2. Capacity Estimation 3. High-Level Design 4. Real-Time Messaging 5. Channel Architecture 6. Message Storage 7. Threads 8. Presence System 9. Search 10. File Sharing 11. Scalability

Chapter 53: Design Slack/Discord

Real-time team collaboration platform supporting channels, DMs, threads, presence, and file sharing at scale.

1. Requirements

Functional Requirements

Feature Description Priority
Real-time messaging Send/receive messages instantly (<100ms latency) P0
Channels Public/private channels, channel creation/deletion, membership management P0
Direct messages 1:1 and group DMs (up to 9 participants) P0
Threads Reply to messages in thread context, thread followers P0
Reactions Emoji reactions on messages, reaction counts P1
File sharing Upload files (up to 1GB), preview generation, inline display P0
Presence Online/away/DND status, typing indicators P1
Search Full-text search across messages, filters (from, in, before/after) P1
Message history Infinite scroll, pagination, message editing/deletion P0
Notifications Push notifications, @mentions, channel-wide alerts P1

Non-Functional Requirements

Requirement Target Justification
Availability 99.9% uptime Team productivity depends on availability
Latency <100ms for message delivery Real-time collaboration requires instant feedback
Consistency Eventual consistency (acceptable for chat) Message ordering per channel, eventual across channels
Scalability 20M DAU, 100K concurrent per server Support enterprise scale
Durability Zero message loss Messages are critical business records

Scale Targets

Metric Value
Daily Active Users (DAU) 20 million
Concurrent connections 5 million (25% of DAU)
Messages per day 500 million
Channels per workspace Average 200, max 500K
Members per channel Average 50, max 100K
Interview Callout: Clarify Requirements
Ask: "What's the max channel size?" (impacts fan-out), "Do we need message ordering guarantees?" (affects architecture), "What's acceptable latency for search?" (determines indexing strategy).

2. Capacity Estimation

Traffic Estimates

Metric Calculation Result
Messages per second 500M messages/day ÷ 86400 sec ~6K writes/sec
Peak writes/sec 6K × 5 (peak factor) 30K writes/sec
Channel fan-out 6K msg/sec × 50 avg members 300K deliveries/sec
Reads per second 10:1 read:write ratio 60K reads/sec
WebSocket connections 5M concurrent / 100K per server 50 gateway servers

Storage Estimates

Component Size per Item Volume Total Storage
Message metadata 500 bytes 500M/day × 365 days 91 TB/year
Message body 1 KB avg 500M/day × 365 days 183 TB/year
File attachments 2 MB avg 50M files/day × 365 days 36 PB/year
Search index 2 KB per message 500M/day × 365 days 365 TB/year
Total (excluding files) - - ~650 TB/year

Bandwidth Estimates

Operation Calculation Bandwidth
Message ingress 6K msg/sec × 1.5 KB 9 MB/sec
Message egress (fan-out) 300K deliveries/sec × 1.5 KB 450 MB/sec
File uploads 50M files/day × 2 MB ÷ 86400 sec 1.2 GB/sec
File downloads File uploads × 3 (download ratio) 3.6 GB/sec
Interview Callout: Back-of-envelope Math
Show the fan-out calculation clearly: "1 message to 50-member channel = 50 deliveries via pub/sub. 6K msg/sec becomes 300K deliveries/sec. This drives WebSocket gateway sizing."

3. High-Level Design

Architecture Overview

graph TB Client[Client Apps
Web/Mobile/Desktop] subgraph "Edge Layer" LB[Load Balancer] CDN[CDN
Static Assets] end subgraph "Gateway Layer" WSG1[WebSocket Gateway 1
100K connections] WSG2[WebSocket Gateway 2] WSGN[WebSocket Gateway N] end subgraph "API Layer" API[REST API
Auth, Channel Mgmt] Auth[Auth Service] end subgraph "Core Services" MsgSvc[Message Service
Publish/Route] ChanSvc[Channel Service
Membership/ACL] PresSvc[Presence Service
Online/Away] NotifSvc[Notification Service
Push/Email] FileSvc[File Service
Upload/Preview] SearchSvc[Search Service
Query/Index] end subgraph "Data Stores" Redis[(Redis
Presence/Cache)] MySQL[(Vitess/MySQL
Messages/Channels)] ES[(Elasticsearch
Search Index)] S3[(S3
File Storage)] end subgraph "Messaging Infrastructure" Kafka[Kafka
Message Queue] PubSub[Redis Pub/Sub
Channel Delivery] end Client -->|HTTPS| LB Client -->|WebSocket| LB LB --> API LB --> WSG1 & WSG2 & WSGN WSG1 & WSG2 & WSGN --> MsgSvc WSG1 & WSG2 & WSGN --> PresSvc WSG1 & WSG2 & WSGN -.subscribe.-> PubSub API --> Auth API --> ChanSvc API --> FileSvc API --> SearchSvc MsgSvc --> Kafka MsgSvc --> PubSub MsgSvc --> MySQL ChanSvc --> MySQL PresSvc --> Redis SearchSvc --> ES FileSvc --> S3 NotifSvc --> Kafka Kafka --> SearchSvc Kafka --> NotifSvc

Component Responsibilities

Component Responsibility Technology
WebSocket Gateway Maintain persistent connections, route messages, subscribe to channels Node.js/Go with Socket.IO
Message Service Validate, persist, publish messages to pub/sub Java/Go microservice
Channel Service Manage channels, membership, ACL checks Java microservice
Presence Service Track online status, heartbeats, typing indicators Go with Redis
Notification Service Process @mentions, send push notifications Consumer from Kafka
Search Service Index messages, execute queries Elasticsearch wrapper
File Service Upload to S3, generate previews, virus scan Java/Python

Data Flow Summary

Flow Path Latency
Send message Client → WS Gateway → Message Service → MySQL + Pub/Sub → WS Gateway → Clients <100ms
Load history Client → API → Message Service → MySQL (paginated) <200ms
Search Client → API → Search Service → Elasticsearch <500ms
File upload Client → File Service → S3 → Message Service (file URL) 1-5 sec
Presence update Client → WS Gateway → Presence Service → Redis → Fan-out to shared channels <50ms

4. Real-Time Messaging

WebSocket Connection Management

Aspect Design Decision Rationale
Protocol WebSocket (wss://) Full-duplex, low latency, push from server
Connection state In-memory on gateway server Fast lookup, no external dependency
Connection affinity Sticky sessions via LB (IP hash) Keep user on same gateway server
Heartbeat 30 sec ping/pong Detect dead connections, update presence
Reconnect Exponential backoff (1s, 2s, 4s, max 30s) Handle temporary network issues
Missed messages Client sends last_msg_id on reconnect Fetch gap from message service

Message Routing Architecture

Step Component Action
1 Client Send message via WebSocket: {channel_id, text, thread_id?}
2 WS Gateway Forward to Message Service via HTTP/gRPC
3 Message Service Validate (ACL check), generate message ID (Snowflake)
4 Message Service Persist to MySQL (sharded by channel_id)
5 Message Service Publish to Redis Pub/Sub: channel:{channel_id}
6 Message Service Enqueue to Kafka (async processing: search, notifications)
7 All WS Gateways Receive from Redis Pub/Sub, fan out to connected clients subscribed to channel

Pub/Sub Channel Subscription Model

Event Action Redis Pub/Sub Topic
User connects WS Gateway subscribes to all user's channels channel:{channel_id} (N subscriptions)
Join channel WS Gateway adds subscription dynamically channel:{new_channel_id}
Leave channel WS Gateway removes subscription Unsubscribe from channel:{channel_id}
Message arrives WS Gateway pushes to all connected clients in channel Fan-out within gateway
User disconnects WS Gateway unsubscribes from all channels Cleanup subscriptions

Message Send Sequence

sequenceDiagram participant C1 as Client 1 participant WSG as WS Gateway participant MS as Message Service participant DB as MySQL participant PS as Redis Pub/Sub participant WSG2 as WS Gateway 2 participant C2 as Client 2 C1->>WSG: Send message (channel_id, text) WSG->>MS: POST /messages {channel_id, user_id, text} MS->>MS: Validate ACL, generate msg_id MS->>DB: INSERT message (sharded by channel_id) DB-->>MS: ACK MS->>PS: PUBLISH channel:123 {msg_id, user_id, text, ts} MS-->>WSG: 201 Created {msg_id, ts} WSG-->>C1: ACK {msg_id, ts} PS-->>WSG: Message on channel:123 WSG->>C1: Push message (subscribed to channel) PS-->>WSG2: Message on channel:123 WSG2->>C2: Push message (subscribed to channel) Note over MS: Async: Enqueue to Kafka for search indexing, notifications
Interview Callout: Why Pub/Sub + Kafka?
"Redis Pub/Sub for real-time fan-out (low latency). Kafka for async processing (search indexing, notifications) with durability. Pub/Sub is ephemeral—if a gateway is down, Kafka ensures no message is lost for indexing."

5. Channel Architecture

Channel Types

Type Characteristics Use Case
Public channel Discoverable, anyone in workspace can join #general, #engineering, #random
Private channel Invite-only, not discoverable #leadership, #hr-confidential
Direct message (1:1) Two participants, always private DM between Alice and Bob
Group DM Up to 9 participants, private, no admin Quick team discussion

Channel Membership Schema

Table Columns Index
channels channel_id (PK), workspace_id, name, type (public/private), created_at, created_by IDX on workspace_id
channel_members channel_id, user_id, role (admin/member), joined_at, last_read_msg_id IDX on (user_id, channel_id), IDX on (channel_id)
channel_metadata channel_id, member_count, message_count, last_message_at For channel list display

Fan-Out Strategies

Channel Size Strategy Implementation
Small (<100) Fan-out on write Pub/Sub to channel:{id}, all gateways push to connected members
Medium (100-1000) Fan-out on write (same) Redis Pub/Sub scales to 10K channels easily
Large (1K-10K) Fan-out on write with batching Gateway batches WebSocket sends (100ms window, 50 msg batch)
Mega (10K+) Hybrid: fan-out on read for offline users Only push to online users (via presence check). Offline users fetch on reconnect.

Large Channel Optimization

Problem Solution Benefit
100K members, 50K online Check presence before fan-out Only push to 50K, not 100K
Redis Pub/Sub saturation Dedicated Pub/Sub cluster for mega channels Isolate traffic
Message rate spike (100/sec) Rate limit + throttle display (show "N new messages") Prevent client overload
Offline members missing messages Store unread count per user, fetch on reconnect Catch up on history

Channel ACL Checks

Operation Check Cache
Send message User is member of channel Redis: user:{user_id}:channels (Set)
Read history User is member of channel Same cache
Join channel Channel is public OR user is invited Invalidate cache after join
Delete message User is message author OR channel admin Check on-demand (rare operation)
Interview Callout: Mega Channel Tradeoff
"For 100K-member channels (e.g., company-wide #general), fan-out on write becomes expensive. Hybrid approach: push to online users (~25% = 25K), offline users pull on reconnect. Tradeoff: online users get instant updates, offline users have slight delay."

6. Message Storage

Message Schema

Column Type Description
message_id BIGINT (PK) Snowflake-style ID (time-sortable)
channel_id BIGINT Shard key (messages partitioned by channel)
user_id BIGINT Message author
thread_id BIGINT (NULL) Parent message ID (NULL for top-level)
text TEXT Message content (up to 40K chars)
attachments JSON Array of file URLs/metadata
reactions JSON {"thumbs_up": 5, "heart": 3}
created_at TIMESTAMP Message timestamp
edited_at TIMESTAMP (NULL) Last edit time
deleted_at TIMESTAMP (NULL) Soft delete (for audit trail)

Snowflake ID Structure

Bits Component Purpose
1 bit Sign (unused) Always 0
41 bits Timestamp (ms since epoch) Time-sortable (69 years range)
10 bits Machine ID 1024 message service instances
12 bits Sequence number 4096 IDs per ms per machine

Benefit: IDs are globally unique, time-sortable, no coordination required. Supports pagination by ID.

Sharding Strategy

Shard Key Pros Cons Decision
channel_id All messages for channel on same shard (fast queries) Hot channels create hot shards ✓ Use this
message_id Even distribution Channel queries scatter across shards ✗ Too slow
user_id Good for "my messages" queries Channel queries scatter ✗ Not primary use case

Handling hot shards: Monitor shard load. Split mega-channels across multiple shards (e.g., shard by hash(channel_id + date)).

Message Pagination

Direction Query Use Case
Load recent SELECT * FROM messages WHERE channel_id = ? ORDER BY message_id DESC LIMIT 50 Initial channel load
Load older WHERE channel_id = ? AND message_id < ? ORDER BY message_id DESC LIMIT 50 Scroll up (infinite scroll)
Load newer WHERE channel_id = ? AND message_id > ? ORDER BY message_id ASC LIMIT 50 Scroll down or catch up after disconnect
Jump to message WHERE channel_id = ? AND message_id >= ? LIMIT 25 + fetch 25 before Jump to thread or search result

Append-Only Optimization

Optimization Implementation Benefit
Write-optimized Messages never updated (edits stored as metadata) No locks, pure INSERTs
Soft deletes Set deleted_at timestamp Audit trail, compliance
Compression gzip text for messages >7 days old Reduce storage cost
Archival Move messages >1 year to cold storage (S3 Glacier) 99% of queries are recent messages
Interview Callout: Why Channel-based Sharding?
"Slack's primary query is 'show me channel history.' Sharding by channel_id keeps all messages on one shard, single-query pagination. Hot channels (e.g., #general) can be split by date if needed."

7. Threads

Thread Data Model

Component Implementation Storage
Parent message Regular message with thread_id = NULL messages table
Thread reply Message with thread_id = parent_message_id messages table
Thread metadata reply_count, last_reply_at, participant_ids[] thread_metadata table (denormalized)
Thread followers thread_id, user_id, following (bool) thread_followers table

Thread Queries

Query SQL Index
Fetch thread SELECT * FROM messages WHERE thread_id = ? ORDER BY message_id ASC IDX on (thread_id, message_id)
Thread count SELECT reply_count FROM thread_metadata WHERE thread_id = ? Cached in Redis
Active threads SELECT * FROM thread_metadata WHERE channel_id = ? ORDER BY last_reply_at DESC LIMIT 20 IDX on (channel_id, last_reply_at)
My threads SELECT * FROM thread_followers WHERE user_id = ? AND following = TRUE IDX on (user_id, following)

Thread Notifications

Condition Notify Who Implementation
Reply to thread All thread followers Query thread_followers, send push notification
@mention in thread Mentioned user (even if not following) Parse mentions, send direct notification
First reply Parent message author (auto-follow) Insert into thread_followers
Unfollow thread N/A Update following = FALSE (don't delete row for analytics)

Thread UI Patterns

Pattern Display Data Loaded
Thread preview Show "5 replies" link below parent message Thread metadata (reply_count, last_reply_at)
Thread sidebar Open thread in right panel, show all replies Full thread messages (paginated if >100 replies)
Thread badge Unread count on thread ("3 new replies") Compare last_read_msg_id vs. last_reply_id
Thread list "All Threads" view, sorted by last_reply_at Thread metadata + parent message preview
Interview Callout: Thread Followers Auto-Join
"When you reply to a thread, you auto-follow it (get notifications for future replies). This is stored in thread_followers. Prevents spam: if you unfollow, you won't get notified unless @mentioned."

8. Presence System

Presence States

State Trigger Duration
Online Active WebSocket connection, heartbeat within 30s While connected
Away No activity for 5 minutes (no keypress, no message sent) Until activity resumes
Do Not Disturb User sets DND (suppress notifications) User-configured (e.g., 2 hours)
Offline WebSocket disconnected, no heartbeat for >60s Until reconnect

Presence Storage

Data Structure Redis Key Value TTL
User presence presence:{user_id} {status: "online", last_seen: 1672531200} 60 seconds
Typing indicator typing:{channel_id} Set of user_id (SADD/SREM) 5 seconds per user
Connection state conn:{user_id} {gateway_id, conn_id, connected_at} No TTL (cleaned on disconnect)

Heartbeat Mechanism

Action Frequency Implementation
Client ping Every 30 seconds WebSocket PING frame
Server pong Immediate WebSocket PONG frame
Presence update On every heartbeat Redis SET presence:{user_id} with 60s TTL
Stale detection 60s timeout Redis key expires → user marked offline

Presence Fan-Out

Event Fan-Out To Implementation
User goes online All users in shared channels Query channel_members, send presence update via Pub/Sub
User goes offline All users in shared channels Same fan-out
Typing indicator Only users viewing the same channel Pub/Sub to channel:{channel_id}:typing
Status change (DND) All users in shared channels Same as online/offline

Typing Indicator Flow

sequenceDiagram participant C as Client participant WSG as WS Gateway participant PS as Presence Service participant Redis as Redis participant PSub as Redis Pub/Sub participant C2 as Other Clients C->>WSG: Typing start (channel_id) WSG->>PS: POST /presence/typing {user_id, channel_id} PS->>Redis: SADD typing:channel123 user456 (TTL 5s) PS->>PSub: PUBLISH channel:123:typing {user_id: 456, typing: true} PSub-->>WSG: Typing event WSG->>C2: Push typing indicator Note over C: User stops typing (no events for 3s) Redis->>Redis: Key expires (TTL 5s) PS->>PSub: PUBLISH channel:123:typing {user_id: 456, typing: false} PSub-->>WSG: Typing event WSG->>C2: Clear typing indicator

Optimization: Presence Bloom Filter

Problem Solution Benefit
Large channels: 10K users, need to check who's online Bloom filter per channel in Redis Fast membership check (O(1) vs. O(N))
False positives ("user might be online") Fallback to exact check in Redis hash Rare, acceptable tradeoff
Stale bloom filter Rebuild every 60 seconds Matches TTL of presence keys
Interview Callout: Why Not Fan Out Presence to Everyone?
"Naive approach: broadcast user status to all workspace members. Problem: 10K workspace = 10K updates per status change. Solution: only fan out to shared channels. If Alice and Bob don't share channels, Bob doesn't get Alice's presence updates."

10. File Sharing

File Upload Flow

Step Component Action
1 Client Request upload URL: POST /files/upload-url {filename, size, mime_type}
2 File Service Generate presigned S3 URL (valid for 15 min)
3 Client Upload file directly to S3 via presigned URL (bypass server)
4 S3 Trigger Lambda on upload (event notification)
5 Lambda Virus scan (ClamAV), generate preview (images/PDFs)
6 Lambda Write metadata to files table (file_id, url, size, status)
7 Client Send message with file attachment: {text, file_ids: [...]}
8 Message Service Store message with file references

File Storage Schema

Column Type Description
file_id BIGINT (PK) Unique file identifier
workspace_id BIGINT Tenant isolation
uploaded_by BIGINT User who uploaded
filename VARCHAR(255) Original filename
mime_type VARCHAR(100) image/png, application/pdf, etc.
size_bytes BIGINT File size (max 1GB)
s3_key VARCHAR(512) S3 object key
preview_url VARCHAR(512) Thumbnail/preview (NULL if not applicable)
status ENUM pending, scanning, ready, quarantined
created_at TIMESTAMP Upload timestamp

File Preview Generation

File Type Preview Strategy Tool
Images Generate 300x300 thumbnail + 1200x1200 preview ImageMagick / Sharp.js
PDFs Render first page as image (300 DPI) Ghostscript / pdf2image
Videos Extract frame at 1 sec + GIF preview (3 sec loop) FFmpeg
Text/Code Syntax-highlighted preview (first 50 lines) Prism.js
Office docs Convert to PDF, then generate image preview LibreOffice / Google Docs API
Archives (zip) List contents (filenames only) unzip -l

Virus Scanning

Step Action Tool
1 Lambda triggered on S3 upload S3 event notification
2 Download file from S3 (streaming) AWS SDK
3 Scan with ClamAV (or third-party API) ClamAV / VirusTotal API
4 If clean: set status = "ready" Update files table
5 If infected: set status = "quarantined", delete from S3 Notify uploader via DM

File Access Control

Scenario Access Rule Implementation
File in public channel All workspace members can access Check workspace membership
File in private channel Only channel members can access Check channel membership via channel_members
File in DM Only DM participants can access Check DM participant list
File link shared externally Generate time-limited signed URL (1 hour) S3 presigned URL with expiry
Interview Callout: Why Presigned URLs?
"Direct S3 upload via presigned URL avoids routing 1GB files through application servers. Reduces latency, saves bandwidth. File Service only handles metadata, not file bytes."

11. Scalability

WebSocket Gateway Scaling

Metric Target Strategy
Connections per server 100K connections Node.js with cluster mode (8 cores × 12.5K conn/core)
Gateway count 50 servers for 5M concurrent Horizontal scaling (auto-scale based on conn count)
Load balancing Sticky sessions (IP hash) AWS ALB with stickiness enabled
Memory per connection ~10 KB 100K × 10 KB = 1 GB per server
Connection failover Client auto-reconnects Exponential backoff, fetch missed messages

Database Sharding Strategy

Data Type Shard Key Sharding Tool
Messages channel_id Vitess (MySQL sharding)
Channels workspace_id Vitess
Users user_id Vitess
Workspaces No sharding (small table) Single MySQL instance with replicas

Workspace Isolation

Component Isolation Strategy Benefit
Database Shard by workspace_id Large workspaces don't impact small ones
Redis Pub/Sub Dedicated cluster for mega workspaces (>10K users) Prevent noisy neighbor problem
Elasticsearch One index per workspace Independent scaling, easier to migrate
S3 files Bucket prefix by workspace: s3://bucket/workspace123/files/ Quota enforcement, cost tracking

Caching Strategy

Data Cache Location TTL Invalidation
Channel membership Redis: user:{user_id}:channels 10 min On join/leave channel
Recent messages Redis: channel:{channel_id}:recent (last 50) 5 min On new message
User profile Redis: user:{user_id}:profile 30 min On profile update
Workspace metadata Redis: workspace:{workspace_id}:meta 1 hour Rare updates
Search results Redis: search:{query_hash} 5 min Time-based (no invalidation)

Rate Limiting

Operation Limit Window Enforcement
Send message 100 messages Per minute Redis counter per user
Create channel 10 channels Per hour Redis counter per user
File upload 50 files Per hour Redis counter per user
Search query 20 queries Per minute Redis counter per user
API calls 1000 requests Per minute API Gateway rate limiter

Monitoring & Metrics

Metric Target Alert Threshold
Message latency (p99) <100ms >200ms
WebSocket connection count 5M concurrent >80% capacity per server
Pub/Sub lag <50ms >500ms
Search query latency (p95) <500ms >2 sec
Database replication lag <1 sec >5 sec
File upload success rate >99% <95%
Interview Callout: Bottleneck Analysis
"Initial bottleneck: database writes (6K msg/sec). Solution: shard by channel_id, use Vitess. Next bottleneck: Redis Pub/Sub fan-out (300K deliveries/sec). Solution: cluster Pub/Sub, dedicate cluster for mega channels. Final bottleneck: WebSocket gateway capacity (5M connections). Solution: horizontal scaling (50 servers @ 100K conn each)."

Summary

Component Technology Choice Key Insight
Real-time delivery WebSocket + Redis Pub/Sub Sub-100ms latency for message fan-out
Message storage Vitess (MySQL sharded by channel_id) Append-only, time-sortable Snowflake IDs
Channel fan-out Fan-out on write (Pub/Sub to all gateways) Hybrid for mega channels (only push to online users)
Presence Redis with TTL (60s heartbeat) Fan out only to shared channels
Search Elasticsearch (one index per workspace) Async indexing via Kafka (1-5 sec lag acceptable)
File sharing S3 with presigned URLs Direct upload bypasses app servers
Scalability 100K connections per gateway server Workspace isolation prevents noisy neighbors

Extended Topics for Deep Dive