Chapter 3: Networking Deep Dive

Introduction

Network behavior is the foundation of distributed systems. Most system design failures trace back to misunderstanding network fundamentals: TCP mechanics, connection lifecycle, protocol overhead, and failure modes.

This chapter covers the critical networking concepts you need for both building production systems and acing FAANG interviews. We go deep on Layer 4 vs Layer 7 proxying, connection pooling strategies, HTTP keep-alive mechanics, head-of-line blocking across protocols, and data center networking topologies.

L4: Transport Layer Networking

Layer 4 operates at the TCP/UDP level. It sees ports, sequence numbers, and connection state, but not application protocols like HTTP.

TCP vs UDP Multiplexing

Both TCP and UDP multiplex multiple application connections over a single network interface using port numbers.

Socket Buffers & Backlog Queue

Each TCP connection has kernel-managed buffers for sending and receiving data. The accept backlog queue holds completed connections waiting for application to accept().

SYN Flood Protection: SYN Cookies

SYN flood attacks send many SYN packets without completing the handshake, exhausting the SYN backlog. SYN cookies encode connection state in the sequence number, avoiding memory allocation.

TIME_WAIT State and Its Implications

When a TCP connection closes, the side that initiates the close (sends FIN first) enters TIME_WAIT state for 2*MSL (Maximum Segment Lifetime, typically 60-120 seconds). This prevents delayed packets from old connections from being misinterpreted.

SO_REUSEPORT: Load Balancing Across Processes

SO_REUSEPORT (Linux 3.9+) allows multiple processes to bind to the same port. The kernel distributes incoming connections across them, enabling per-process accept queues.

L7: Application Layer Networking

Layer 7 operates at the application protocol level (HTTP, gRPC, etc.). It can parse headers, route based on URL paths, terminate SSL, and cache responses.

HTTP Parsing and Header Inspection

L7 proxies must parse the entire HTTP request to make routing decisions. This adds CPU cost and latency compared to L4.

SSL Termination at L7

L7 proxies can terminate TLS, decrypt traffic, inspect it, then re-encrypt to backends. This enables caching, compression, and WAF (Web Application Firewall) inspection.

Content-Based Routing

L7 proxies route based on HTTP headers, URL paths, cookies, or even request body content.

L4 vs L7 Load Balancing: Deep Comparison

This is a critical system design decision. L4 is faster but less flexible; L7 is feature-rich but adds overhead.

Feature Comparison

Direct Server Return (DSR)

With DSR, the load balancer forwards requests to backends, but backends respond directly to clients, bypassing the LB. This asymmetric routing avoids LB becoming a bottleneck for response traffic.

Connection Pooling

Connection pooling reuses TCP connections instead of opening a new one for each request. This avoids the latency cost of TCP + TLS handshakes.

Why Pool Connections?

Pool Sizing Formulas

Pool size is a critical tuning parameter. Too small: requests wait for connections. Too large: memory waste, backend overload.

HTTP Connection Pools

HTTP clients (curl, requests, httpx, OkHttp) maintain connection pools per destination host.

# Python httpx with connection pooling
import httpx

# Pool with 100 max connections, 10 per host
client = httpx.Client(
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=10)
)

# Reuses connections automatically
response1 = client.get("https://api.example.com/users")
response2 = client.get("https://api.example.com/posts")  # Same connection!

Database Connection Pools

Database connections are expensive (authentication, session state). Pools are essential for databases.

Keep-Alive: HTTP vs TCP

Keep-alive has two meanings: HTTP keep-alive (connection reuse) and TCP keepalive (dead connection detection). They're often confused!

HTTP Keep-Alive (Connection Reuse)

HTTP/1.0 closed connections after each request. HTTP/1.1 defaults to persistent connections (Connection: keep-alive), allowing multiple requests per TCP connection.

TCP Keepalive (Connection Liveness Probe)

TCP keepalive is a mechanism to detect dead connections. After a connection is idle for tcp_keepalive_time, the kernel sends a probe packet. If no response after multiple probes, the connection is closed.

HTTP Keep-Alive Timeout Tuning

Servers close idle keep-alive connections after a timeout. Too short: clients must reconnect frequently. Too long: server holds many idle connections.

Head-of-Line (HOL) Blocking

HOL blocking occurs when one slow/lost packet blocks all subsequent packets, even if they're independent. This manifests at multiple layers.

TCP Head-of-Line Blocking

TCP guarantees in-order delivery. If packet N is lost, packets N+1, N+2, ... are buffered (not delivered to application) until N is retransmitted.

HTTP/1.1 Head-of-Line Blocking

HTTP/1.1 sends requests serially over a connection. Request 2 cannot be sent until response 1 is received, even if they're independent.

HTTP/2 Stream Multiplexing (Solves L7 HOL, Not L4)

HTTP/2 multiplexes multiple streams over a single TCP connection. Each stream is independent, so one slow response doesn't block others.

However, HTTP/2 still suffers from TCP-level HOL blocking. If one TCP packet is lost, all streams on that connection wait for retransmission.

HTTP/3 & QUIC: True Multiplexing Without HOL Blocking

QUIC (HTTP/3) runs over UDP and implements per-stream reliability. Loss on one stream doesn't block others.

Connection Management at Scale

At scale, connection limits become bottlenecks: file descriptors, ephemeral ports, connection tracking tables.

File Descriptor Limits

On Unix systems, each socket is a file descriptor. Processes have per-process and system-wide limits.

# Check per-process limit
ulimit -n
# Output: 1024 (default on many systems)

# Check system-wide limit
cat /proc/sys/fs/file-max
# Output: 1048576 (example)

# Increase per-process limit (requires root or capability)
ulimit -n 65536

# Permanent change: edit /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

Ephemeral Port Exhaustion

Clients use ephemeral ports (32768-60999 on Linux) for outbound connections. With TIME_WAIT, ports are held for 60s after close.

Connection Tracking Tables (conntrack)

Linux stateful firewalls (iptables with connection tracking) maintain a table of all connections. This table has a maximum size.

# Check current conntrack entries
cat /proc/sys/net/netfilter/nf_conntrack_count
# Output: 12543

# Check max conntrack entries
cat /proc/sys/net/netfilter/nf_conntrack_max
# Output: 65536

# Increase limit (requires root)
sysctl -w net.netfilter.nf_conntrack_max=262144

TIME_WAIT Accumulation at Scale

High-traffic servers accumulate many TIME_WAIT connections. This is usually harmless (they consume minimal memory), but can be confusing.

# Count connections in each state
netstat -an | awk '{print $6}' | sort | uniq -c | sort -rn
# Output:
#   45123 TIME_WAIT
#    8765 ESTABLISHED
#     234 LISTEN
#      12 FIN_WAIT1

Network Protocols for Microservices

Microservices use specialized protocols optimized for service-to-service communication: binary encoding, multiplexing, schema enforcement.

gRPC: HTTP/2-Based RPC

gRPC uses HTTP/2 for transport, Protocol Buffers for serialization, and supports streaming. It's designed for low-latency, high-throughput microservices.

Apache Thrift: Facebook's RPC Framework

Thrift is similar to gRPC: schema-defined, binary encoding, cross-language. Predates gRPC (2007 vs 2015), widely used at Facebook/Meta.

Protocol Buffers Wire Format

Protobuf uses variable-length encoding and field tags for efficient binary serialization.

// Protobuf schema
message User {
  int32 id = 1;
  string name = 2;
  repeated string emails = 3;
}

// Example: User{id: 123, name: "Alice", emails: ["a@example.com"]}
// Wire format (hex):
// 08 7B       - field 1 (id), varint, value 123
// 12 05 41 6C 69 63 65  - field 2 (name), length-delimited, "Alice"
// 1A 0F 61 40 65 78 61 6D 70 6C 65 2E 63 6F 6D  - field 3 (emails[0]), "a@example.com"
// Total: 29 bytes

// Equivalent JSON:
// {"id":123,"name":"Alice","emails":["a@example.com"]}
// Total: 52 bytes (1.8x larger)

Service Mesh Networking

Service meshes (Istio, Linkerd, Consul Connect) add a networking layer between microservices, providing observability, security, and traffic control without changing application code.

Sidecar Proxy Model (Envoy)

Each service pod has a sidecar proxy (Envoy) that intercepts all inbound/outbound traffic. The proxy enforces policies, collects metrics, and handles retries.

Traffic Routing in Service Mesh

Service mesh proxies route traffic based on rules: canary deployments, A/B tests, fault injection.

Circuit Breaking at Network Level

Envoy can circuit-break at the connection level: if backend has too many connections or errors, stop sending traffic.

Mutual TLS (mTLS) Between Services

Service mesh enables zero-trust networking: every service-to-service call uses mTLS for authentication and encryption.

Network Debugging

When systems fail, network issues are often the culprit. Knowing how to diagnose connection problems is essential.

tcpdump & Wireshark Patterns

tcpdump captures raw packets. Wireshark provides GUI analysis. Key patterns to recognize:

# Capture packets on interface eth0, port 80
tcpdump -i eth0 -n port 80

# Capture full packet content (not just headers)
tcpdump -i eth0 -n -A port 80

# Save to file for Wireshark analysis
tcpdump -i eth0 -w capture.pcap port 80

# Common patterns:

# 1. SYN retransmissions (server not responding)
# Client sends SYN, no SYN-ACK, client retries
12:00:00.000 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [S], seq 123
12:00:01.000 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [S], seq 123  # Retransmit!
12:00:03.000 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [S], seq 123  # Retransmit!

# 2. RST from server (connection refused)
12:00:00.000 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [S], seq 123
12:00:00.001 IP 10.0.0.5.80 > 1.2.3.4.5678: Flags [R], seq 0  # Reset!

# 3. Window size 0 (receiver buffer full, flow control)
12:00:00.000 IP 10.0.0.5.80 > 1.2.3.4.5678: Flags [.], ack 500, win 0
# Sender must wait for window update

# 4. Duplicate ACKs (packet loss, triggers fast retransmit)
12:00:00.000 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [.], ack 100
12:00:00.001 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [.], ack 100  # Duplicate
12:00:00.002 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [.], ack 100  # Duplicate
12:00:00.003 IP 1.2.3.4.5678 > 10.0.0.5.80: Flags [.], ack 100  # 3rd duplicate = retransmit

Network Partitions: How They Look

A network partition splits a cluster into two groups that cannot communicate. Each group may think the other is dead.

Split-Brain Scenarios

Split-brain occurs when a cluster partitions and each side elects a leader, causing divergent state.

Bandwidth vs Latency vs Throughput

These terms are often confused. Understanding the distinctions is critical for capacity planning.

Definitions

Bandwidth-Delay Product (BDP)

BDP is the amount of data "in flight" on a network link. It determines optimal TCP buffer size.

Buffer Bloat: Too Much Buffering

Bufferbloat occurs when network devices have excessively large buffers, increasing latency under load.

Data Center Networking

Modern datacenters use specialized topologies to provide high bandwidth, low latency, and fault tolerance for distributed systems.

Spine-Leaf Topology

Spine-leaf is the standard datacenter topology. Every leaf switch connects to every spine switch, providing equal-cost paths between any two servers.

East-West vs North-South Traffic

Equal-Cost Multi-Path (ECMP) Routing

ECMP load balances traffic across multiple equal-cost paths. In spine-leaf, a packet from Leaf 1 to Leaf 2 can go through Spine 1, 2, or 3.

BGP for Anycast: Same IP, Multiple Locations

Anycast announces the same IP prefix from multiple locations. Routers send traffic to the nearest location based on BGP routing.

Interview Preparation: Networking Questions

Common FAANG interview questions and how to answer them with depth.

TCP Congestion Control Algorithms Deep Dive

TCP congestion control prevents overwhelming the network. Different algorithms use different strategies to detect and respond to congestion, with massive performance differences in high-bandwidth or lossy networks.

TCP Reno: The Classic Algorithm

TCP Reno (1990) is the foundational congestion control algorithm. It uses packet loss as the primary signal of congestion.

Four Phases of Reno

AIMD Sawtooth Pattern

Reno's Additive Increase / Multiplicative Decrease (AIMD) creates a characteristic sawtooth graph:

Problem: Reno treats all loss as congestion. In wireless/cellular networks with random loss (signal interference, not congestion), Reno unnecessarily reduces throughput.

CUBIC: High-BDP Optimization

CUBIC (2008) is the default algorithm in Linux since 2.6.19. It solves Reno's inefficiency in high-bandwidth delay product (BDP) networks.

Reno's Problem: Slow Recovery in High-BDP Networks

Example: 1 Gbps link, 100ms RTT → BDP = 12.5 MB. If cwnd was 10,000 packets before loss, Reno cuts to 5,000 packets, then adds 1 packet/RTT. To recover to 10,000 packets takes 5,000 RTTs = 500 seconds (8+ minutes!).

CUBIC's Solution: Cubic Function Growth

Instead of linear growth, CUBIC uses a cubic function centered around the last congestion point:

W(t) = C × (t - K)³ + Wmax

Where:
- W(t) = cwnd at time t after last congestion event
- C = scaling constant (0.4 by default)
- K = time to reach Wmax (when cubic function = 0)
- Wmax = cwnd when last congestion occurred
- t = elapsed time since last congestion

Growth Pattern

• Just after loss: Fast growth (concave phase) to quickly recover
• Near Wmax: Slow growth (inflection point) to cautiously probe if capacity increased
• Beyond Wmax: Moderate growth (convex phase) to find new capacity

Result: CUBIC recovers much faster than Reno in high-BDP networks (data center, long-distance fiber). In the 1 Gbps / 100ms RTT example, CUBIC recovers in seconds instead of 8 minutes.

RTT Fairness

CUBIC's growth is time-based (not RTT-based like Reno). This improves fairness between flows with different RTTs sharing the same bottleneck.

BBR (Bottleneck Bandwidth and RTT): The Modern Approach

BBR (2016) from Google represents a paradigm shift: don't use packet loss as a congestion signal.

Why Loss is a Bad Signal

• Random loss: Wireless networks have 1-5% loss from signal issues, not congestion
• Bufferbloat: Large router buffers hide congestion until buffers fill and loss occurs → high latency before correction
• Reactive: Loss-based algorithms wait for congestion to occur, then react

BBR's Approach: Model-Based Control

BBR builds a model of the network path by continuously measuring:

• BtlBw (Bottleneck Bandwidth): Maximum delivery rate observed in recent window (10 RTTs)
• RTprop (Round-Trip Propagation time): Minimum RTT observed in recent window (10s)

Key Insight: Optimal operating point is cwnd = BtlBw × RTprop (the BDP). At this point, the pipe is full but queues are not building.

Four BBR States

BBR Performance

Real-world results:

• YouTube: 2-25x throughput improvement on mobile/satellite networks with 1-5% loss
• Google B4 network: 2x throughput increase in wide-area data center links
• Bufferbloat: Reduces latency by 10-100x by not filling buffers unnecessarily

Algorithm Comparison

QUIC vs TCP HOL Blocking — Detailed

Head-of-line (HOL) blocking is a critical performance issue where one slow/lost item blocks unrelated items behind it. QUIC solves TCP's HOL blocking problem, which is why HTTP/3 over QUIC significantly outperforms HTTP/2 over TCP on lossy networks.

TCP HOL Blocking: The Problem

TCP guarantees in-order delivery within a single connection. If packet 5 is lost, packets 6-10 (even if received) are held in the TCP receive buffer and not delivered to the application until packet 5 is retransmitted and arrives.

HTTP/2 + TCP: Multiplexed Streams, Single TCP Connection

HTTP/2 multiplexes multiple HTTP requests (streams) over a single TCP connection. This reduces connection overhead (no repeated handshakes) but inherits TCP's HOL blocking:

Key Issue: A single lost packet (Stream B packet 2) blocks ALL streams, even though Stream A packet 3 and Stream C packet 1 arrived successfully. This is TCP-level HOL blocking.

QUIC Solution: Independent Streams

QUIC is a transport protocol built on UDP (to avoid kernel TCP stack limitations). Each QUIC stream has independent flow control and delivery. A lost packet only blocks ITS stream.

HTTP/3 + QUIC: Same Scenario, No Blocking

Result: Stream A and Stream C complete immediately. Only Stream B waits for its lost packet. No cross-stream blocking.

Real-World Impact

Source: Google measured these results deploying HTTP/3 to YouTube and Chrome.

Why QUIC Needed to Be Built on UDP

TCP is implemented in the operating system kernel. Changing TCP behavior (e.g., per-stream delivery) requires OS updates, which take years to deploy (Windows, macOS, Linux, mobile, IoT). UDP is a thin wrapper around IP packets—applications control all behavior. QUIC implements reliability, congestion control, and multiplexing in userspace (browser/app), allowing rapid iteration without OS changes.

HTTP/1.1 vs HTTP/2 vs HTTP/3 HOL Comparison