Chapter 2: Operating System Concepts for Distributed Systems

2.1 Processes vs Threads

Memory Model

Context Switch Costs

IPC Mechanisms Comparison

When to Use What

Real Numbers: Context Switch Benchmark

// Linux 5.15, Intel Xeon Gold 6248R @ 3.0GHz
// Measured with perf stat and lmbench

Process context switch:     ~6.2 μs
Thread context switch:      ~1.8 μs
Goroutine context switch:   ~200 ns (userspace, no syscall)
System call (getpid):       ~120 ns

2.2 I/O Models

Understanding I/O models is critical for building high-performance network services. The evolution from blocking I/O to async I/O solved the C10K problem.

1. Blocking I/O

Cost: One thread per connection. 10,000 connections = 10,000 threads = ~10GB RAM (1MB stack each).

2. Non-blocking I/O

Problem: Busy-waiting wastes CPU cycles. Inefficient for thousands of sockets.

3. I/O Multiplexing (select/poll/epoll)

Key insight: One thread can monitor thousands of sockets. O(1) event retrieval with epoll.

4. Signal-Driven I/O

Rarely used: Signal handling is tricky, limited socket info in signal.

5. Asynchronous I/O (io_uring)

Future of Linux I/O: True async, zero-copy, poll-less mode. 30-40% faster than epoll for high loads.

Performance Comparison

Real Numbers: Benchmark (100K concurrent connections)

// Linux 5.15, wrk benchmark, echo server
// Metric: Requests/sec per CPU core

Blocking I/O (thread-per-conn):     50 req/s (OOM, thrashing)
epoll (Nginx):                      45,000 req/s
io_uring (poll mode):               62,000 req/s (+38%)

// Latency p99
epoll:                              12 ms
io_uring:                           8 ms

io_uring Advanced: Why It's a Game-Changer

Architecture: io_uring uses submission queue (SQ) and completion queue (CQ) in shared memory between kernel and userspace. In SQPOLL mode, the fast path requires ZERO syscalls—the kernel polls the submission queue.

Operations Supported

io_uring handles the entire I/O stack, not just network sockets:

• File I/O: read, write, openat, statx, fallocate
• Network I/O: send, recv, accept, connect, sendmsg, recvmsg
• Advanced: splice, tee, sync_file_range, fsync

Performance Deep Dive

Benchmark: NVMe SSD (Intel Optane), 4K random reads

epoll (Nginx):              1.1M IOPS
io_uring (standard):        1.3M IOPS (+18%)
io_uring (SQPOLL mode):     1.5M IOPS (+36%)

Why 30-40% faster than epoll for high-throughput workloads:

• Eliminates syscall overhead entirely in poll mode (SQPOLL)
• Batches multiple operations in single submission
• Zero-copy with registered buffers (IORING_REGISTER_BUFFERS)
• Kernel-side command chaining (linked operations)

Key Flags & Features

Real-World Adoption

• RocksDB: Uses io_uring for async I/O, 25% improvement in write throughput
• Tokio (Rust): Async runtime with io_uring backend for Linux
• Seastar framework: C++ async framework (used by ScyllaDB) leverages io_uring
• liburing: User-space library for easy io_uring integration

2.3 The C10K and C10M Problems

The Problem: Thread-per-Connection Math

10,000 connections × 1 MB stack per thread = 10 GB RAM
Context switches: 10,000 threads → ~60,000 μs total switch time per round-robin
Kernel scheduler overhead: O(n) or O(log n) depending on algorithm
Cache thrashing: Each thread evicts others' data from L1/L2/L3 caches

Solution: Event-Driven Architecture

Reactor Pattern (epoll/kqueue)

Proactor Pattern (io_uring/IOCP)

Key Differences

C10M Problem (10 Million Connections)

Achieving 10M concurrent connections requires:

• Kernel bypass: DPDK, XDP (process packets in userspace)
• Zero-copy networking: io_uring, sendfile()
• CPU pinning: Avoid cross-CPU migrations
• Huge pages: Reduce TLB misses (2MB pages vs 4KB)
• Lock-free data structures: Per-core queues, RCU
• Memory: ~64GB (10M × 6KB per connection state)

Real Numbers: WhatsApp (2012)

// 50 engineers, 450 million users, 2 million connections per server
// Stack: Erlang/OTP, FreeBSD, kqueue

Connections per server:     2,000,000
Servers:                    ~500
CPU usage:                  ~60% (mostly crypto)
Memory per connection:      ~4 KB
Network I/O model:          kqueue (edge-triggered)

Key: Erlang lightweight processes (300 bytes overhead) + kqueue

2.4 epoll vs kqueue vs IOCP

Deep Comparison

Edge-Triggered vs Level-Triggered

// If 1KB available in socket:
epoll_wait() → socket readable
read(500 bytes)
epoll_wait() → socket readable (still 500 bytes!)
read(500 bytes)
epoll_wait() → no event (buffer empty)

// If 1KB available in socket:
epoll_wait() → socket readable
read(500 bytes)
epoll_wait() → no event (no NEW data)
// Must read() until EAGAIN!

read(500 bytes)
read() → EAGAIN (now fully drained)

Code-Level Differences

Why Nginx Uses epoll

Abstraction Layers

Cross-platform libraries abstract these differences:

Real Numbers: Benchmark Comparison

// Same hardware: Linux 5.15 on Intel Xeon, 100K connections
// Workload: Echo server, 1KB messages

epoll (level-triggered):    38,000 req/s, 15ms p99 latency
epoll (edge-triggered):     45,000 req/s, 12ms p99 latency
io_uring (poll mode):       62,000 req/s, 8ms p99 latency

// FreeBSD 13 on same hardware
kqueue:                     42,000 req/s, 13ms p99 latency

// Windows Server 2022
IOCP:                       50,000 req/s, 10ms p99 latency

2.5 Memory Management for High-Performance Systems

Virtual Memory Architecture

Page Table Lookup Cost

Huge Pages (2MB vs 4KB)

mmap() Use Cases

Copy-on-Write (COW)

NUMA (Non-Uniform Memory Access)

NUMA Best Practices

Real Numbers: Memory Performance

// Intel Xeon Gold 6248R @ 3.0GHz, 128GB RAM, Linux 5.15

// Latency (sequential access)
L1 cache hit:               ~1 ns (4 cycles)
L2 cache hit:               ~3 ns (10 cycles)
L3 cache hit:               ~12 ns (40 cycles)
Local RAM (NUMA):           ~80 ns
Remote RAM (NUMA):          ~140 ns (+75%)
TLB miss (page table walk): ~60 ns

// Huge pages impact (Redis benchmark, 10GB dataset)
4KB pages:                  180,000 ops/sec, 12% TLB miss rate
2MB huge pages:             245,000 ops/sec (+36%), 0.8% TLB miss rate

// Copy-on-Write (fork benchmark)
fork() without COW:         50ms (copy 1GB)
fork() with COW:            0.3ms (just page tables)

2.6 CPU Scheduling and Tail Latency

Completely Fair Scheduler (CFS)

Linux default scheduler since 2.6.23 (2007). Uses red-black tree to track runnable tasks.

Virtual Runtime (vruntime)

vruntime += (actual_runtime * nice_0_weight) / task_weight

// nice_0_weight = 1024 (baseline)
// task_weight depends on nice value (-20 to +19)

Example:
- Task A (nice 0): vruntime += 10ms * 1024 / 1024 = 10ms
- Task B (nice 10): vruntime += 10ms * 1024 / 335 = 30ms
→ Task B gets less CPU share (lower priority)

Scheduling Policies

CPU Affinity (Pinning)

Problem: Context switch across CPUs invalidates L1/L2 caches and TLB.

// Task migrates between CPUs
T0: CPU 0 (warm cache)
T1: CPU 2 (cold cache, 50% slower)
T2: CPU 5 (cold cache again)
T3: CPU 0 (cache refill)

// Task always on CPU 0
taskset -c 0 ./app
// or programmatically:
pthread_setaffinity_np(cpu_set);

Context Switch Costs

Impact on Tail Latency

Scenario: Latency-sensitive service (target p99 < 10ms)

Mitigation Strategies

Real Numbers: Scheduling Impact

// Benchmark: 10K req/s load test, 1ms work per request
// Setup: 16-core Xeon, Linux 5.15

Default CFS (no tuning):
p50: 2ms, p99: 18ms, p99.9: 45ms
Context switches: 25K/sec per core

CPU pinning (taskset -c 0-7):
p50: 2ms, p99: 8ms, p99.9: 15ms (-67% tail)
Context switches: 8K/sec per core

CPU isolation (isolcpus=0-7):
p50: 2ms, p99: 4ms, p99.9: 7ms (-84% tail)
Context switches: 1K/sec per core

SCHED_FIFO (real-time):
p50: 2ms, p99: 3ms, p99.9: 4ms (-91% tail)
Context switches: 200/sec per core
Risk: Misconfiguration can freeze system!

2.7 File Systems for High-Performance Storage

Modern File Systems Comparison

Write Path: Application to Disk

Buffered I/O vs Direct I/O

int fd = open("file.txt", O_RDWR);
write(fd, buf, 4096);
// Returns immediately!
// Data in page cache

int fd = open("file.txt", O_RDWR | O_DIRECT);
// Requires aligned buffers
posix_memalign(&buf, 4096, 4096);
write(fd, buf, 4096);
// Blocks until disk confirms!

Write-Ahead Logging (WAL) and Journaling

Problem: Crash during multi-step operation leaves file system inconsistent.

ext4 Journaling Modes

fsync() Semantics and The PostgreSQL Bug

io_uring for File I/O

Modern async file I/O on Linux (5.1+), replacing aio.

// Traditional sync read
int fd = open("file.txt", O_RDONLY);
read(fd, buf, 4096); // BLOCKS

// io_uring async read
struct io_uring ring;
io_uring_queue_init(256, &ring, 0);

struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, 4096, 0);
io_uring_submit(&ring); // Non-blocking!

// Do other work...

struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe); // Wait for completion
ssize_t bytes_read = cqe->res;

XFS vs ext4 for Databases

Real Numbers: File System Performance

// Benchmark: fio sequential write, 16GB file, Intel Optane SSD
// Setup: Linux 5.15, NVMe drive (7 GB/s capable)

ext4 (buffered I/O):        3.2 GB/s write, 4.5 GB/s read
ext4 (O_DIRECT):            2.8 GB/s write, 4.2 GB/s read
XFS (buffered I/O):         3.5 GB/s write, 4.6 GB/s read
XFS (O_DIRECT):             3.1 GB/s write (+11% vs ext4), 4.4 GB/s read
Btrfs (CoW overhead):       2.1 GB/s write, 4.0 GB/s read
io_uring (XFS):             4.2 GB/s write (+35% vs O_DIRECT), 5.1 GB/s read

// fsync() latency (ext4, ordered mode, SSD)
write() + fsync():          250 μs (metadata + data)
write() + fdatasync():      180 μs (data only, skip metadata)
write() only (no fsync):    2 μs (page cache)

// Page cache hit rate impact (PostgreSQL workload)
50% cache hit rate:         15K TPS
90% cache hit rate:         60K TPS (4x improvement)

2.8 Linux Network Stack

Full Packet Path: NIC to Application

Receive Path Latency Breakdown

Total typical latency: ~18 μs (kernel) + ~5 μs (application) = 23 μs

Zero-Copy Techniques

// Read file, send over network
int fd = open("file.txt", O_RDONLY);
char buf[4096];
read(fd, buf, 4096);    // Copy 1: disk → user
write(sockfd, buf, 4096); // Copy 2: user → kernel

// Zero-copy transfer
int fd = open("file.txt", O_RDONLY);
sendfile(sockfd, fd, NULL, 4096);
// Kernel-only operation!

Zero-Copy APIs

Kernel Bypass: DPDK and XDP

Problem: Kernel network stack adds ~20 μs latency. For ultra-low latency (<5 μs), bypass kernel entirely.

// Pseudo-code
while (1) {
    packets = rte_eth_rx_burst(port, queue, pkts, 32);
    for (i = 0; i < packets; i++) {
        process_packet(pkts[i]);
    }
}
// No interrupts, no context switches!

// eBPF XDP program
SEC("xdp")
int xdp_drop_tcp_syn(struct xdp_md *ctx) {
    // Parse packet, drop SYN floods
    if (is_tcp_syn(ctx)) {
        return XDP_DROP; // No kernel processing!
    }
    return XDP_PASS; // Continue to kernel
}

Receive Side Scaling (RSS)

Distribute incoming packets across multiple CPU cores to avoid single-core bottleneck.

NAPI (New API) Interrupt Mitigation

Problem: At high packet rates (100K+ pps), interrupts overwhelm CPU (IRQ storm).

Real Numbers: Network Stack Performance

// Benchmark: UDP echo server, 64-byte packets
// Setup: Linux 5.15, Intel X710 10GbE NIC, single core

Kernel stack (no tuning):       400K pps, ~25 μs latency
Kernel stack (RSS + NAPI):      1.2M pps, ~20 μs latency
XDP (eBPF filter):              12M pps, ~8 μs latency
DPDK (user-space polling):      14M pps, ~3 μs latency

// Zero-copy benchmark (1GB file transfer over localhost)
Traditional read()/write():     3.2 GB/s, 320 ms
sendfile():                     8.5 GB/s (+165%), 120 ms
io_uring (zero-copy):           11.2 GB/s (+250%), 90 ms

// RSS impact (TCP connections, 10GbE)
Single queue (1 CPU):           800K conn/s, CPU bottleneck
RSS (8 queues, 8 CPUs):         4.5M conn/s, linear scaling

2.9 Containers and Linux Namespaces

Linux Namespaces

Namespaces provide process isolation by virtualizing system resources. Each namespace type isolates a specific aspect of the system.

How Docker Uses Namespaces

cgroups v2 (Control Groups)

cgroups limit and account for resource usage. Critical for multi-tenant systems.

cgroup Enforcement Example

# Create cgroup for container
mkdir /sys/fs/cgroup/container_a
echo $PID > /sys/fs/cgroup/container_a/cgroup.procs

# Limit to 512MB RAM
echo 512M > /sys/fs/cgroup/container_a/memory.max

# Limit to 50% of 2 CPUs (100,000 μs period)
echo "100000 100000" > /sys/fs/cgroup/container_a/cpu.max

# Limit disk writes to 50 MB/s (device 8:0 = /dev/sda)
echo "8:0 wbps=52428800" > /sys/fs/cgroup/container_a/io.max

# What happens on violation?
# - Memory: OOM killer terminates processes in cgroup
# - CPU: Throttled (process sleeps until next period)
# - I/O: Requests queued, throughput capped

OverlayFS: Container Layered File System

Copy-on-Write (CoW) in OverlayFS

Container vs VM Comparison

Real Numbers: Container Performance

// Docker benchmark: Redis throughput
// Setup: Linux 5.15, 16-core Xeon, 64GB RAM

Bare metal:             180K req/s, 0.5ms p99 latency
Docker (default):       175K req/s (-3%), 0.6ms p99 latency
Docker (--net=host):    178K req/s (-1%), 0.5ms p99 latency
VM (KVM):               150K req/s (-17%), 1.2ms p99 latency

// Boot time comparison (hello-world app)
Container (Docker):     80 ms
VM (KVM):               12 seconds
VM (Firecracker):       125 ms (micro-VM)

// Memory overhead (idle)
Container:              8 MB
VM (KVM):               512 MB (minimal guest)
VM (Firecracker):       5 MB (micro-VM)

// Density (16-core host, 64GB RAM)
Containers:             500+ (limited by memory, not overhead)
VMs:                    30-50 (limited by guest OS overhead)

2.10 Concurrency Primitives

Primitives Comparison

Mutex (Mutual Exclusion)

Guarantees: Only one thread can hold the lock at a time.

// C++ std::mutex example
std::mutex mtx;
std::vector<int> shared_data;

void worker() {
    mtx.lock();
    // Critical section: only one thread at a time
    shared_data.push_back(42);
    mtx.unlock();
}

// RAII style (preferred)
void worker_safe() {
    std::lock_guard<std::mutex> lock(mtx);
    shared_data.push_back(42);
    // Automatic unlock on scope exit
}

Semaphore

Use case: Limit concurrent access to N resources (e.g., connection pool of size 10).

// POSIX semaphore example
sem_t pool_semaphore;
sem_init(&pool_semaphore, 0, 10); // 10 available slots

void worker() {
    sem_wait(&pool_semaphore);  // Decrement, block if 0
    // Use resource (e.g., database connection)
    query_database();
    sem_post(&pool_semaphore);  // Increment, wake waiter
}

Spinlock

When to use: Critical section is shorter than context switch cost (~1 μs).

// Spinlock example (Linux kernel style)
spinlock_t lock;
spin_lock_init(&lock);

void critical_function() {
    spin_lock(&lock);
    // Very short critical section (<100 ns)
    counter++;
    spin_unlock(&lock);
}

// WARNING: Holding spinlock too long burns CPU!
// Rule of thumb: Use spinlock if critical section < 100 ns
//                Use mutex if critical section > 1 μs

RWLock (Read-Write Lock)

Optimization: Multiple readers can hold lock simultaneously, but writers get exclusive access.

// C++ shared_mutex (RWLock)
std::shared_mutex rw_lock;
std::map<std::string, int> cache;

// Many readers
void read_cache(const std::string& key) {
    std::shared_lock<std::shared_mutex> lock(rw_lock);
    auto value = cache[key]; // Safe to read
}

// Rare writer
void update_cache(const std::string& key, int value) {
    std::unique_lock<std::shared_mutex> lock(rw_lock);
    cache[key] = value; // Exclusive access
}

Atomics and Compare-and-Swap (CAS)

Lock-free synchronization: Use CPU atomic instructions instead of locks.

// C++ std::atomic example
std::atomic<int> counter(0);

void increment() {
    counter.fetch_add(1, std::memory_order_relaxed);
    // Single CPU instruction (LOCK XADD on x86)
}

// Compare-and-Swap (CAS)
std::atomic<int> value(100);

bool try_decrement() {
    int expected = value.load();
    while (expected > 0) {
        // Atomically: if (value == expected) { value = expected - 1; return true; }
        if (value.compare_exchange_weak(expected, expected - 1)) {
            return true; // Success
        }
        // expected updated to current value, retry
    }
    return false; // Value is 0
}

Memory Ordering (Acquire/Release)

Lock-Free Queue Example

ABA Problem

Problem: CAS can't distinguish between "unchanged" and "changed back to original value".

// ABA Problem Example
// Thread 1: Reads value A from stack top
// Thread 2: Pops A, pops B, pushes A back (stack looks unchanged!)
// Thread 1: CAS succeeds (thinks nothing changed), but stack is corrupted

// Solution 1: Use version counter (ABA → ABA₁ → ABA₂)
struct Node {
    int value;
    uint64_t version; // Incremented on every change
};

// Solution 2: Use hazard pointers (mark nodes in use)
// Solution 3: Use epoch-based reclamation (defer deletion)

When to Use What

Real Numbers: Primitive Performance

// Benchmark: Intel Xeon Gold 6248R @ 3.0GHz, Linux 5.15
// Increment counter 100M times, varying contention

Uncontended (single thread):
  Mutex:                    ~25 ns per lock/unlock
  Spinlock:                 ~5 ns per lock/unlock
  Atomic (relaxed):         ~2 ns per fetch_add
  Atomic (seq_cst):         ~5 ns per fetch_add

High contention (16 threads):
  Mutex:                    ~1.2 μs per lock/unlock (context switch)
  Spinlock:                 ~400 ns per lock/unlock (spin time)
  Atomic CAS (retry loop):  ~50 ns per successful CAS
  RWLock (90% reads):       ~80 ns per read, ~1.5 μs per write

Lock-free queue (SPSC):     80M ops/sec
Lock-free queue (MPMC):     25M ops/sec (more CAS retries)
Mutex-based queue:          5M ops/sec (lock overhead)

2.11 Event Loop Architectures

1. Single-Threaded Event Loop (Node.js)

Characteristics:

• Pros: No thread synchronization overhead, simple mental model, efficient for I/O-bound workloads
• Cons: One CPU-heavy task blocks all requests, cannot utilize multiple cores (without clustering)
• Used by: Node.js, Redis, Nginx (per worker)

2. Multi-Reactor Pattern (Netty, Nginx)

Characteristics:

• Pros: Scales to multiple cores, load balanced, each worker independent
• Cons: Inter-worker communication requires synchronization, uneven load distribution possible
• Used by: Nginx (multi-process), Netty, HAProxy

3. Work-Stealing (Go Runtime, Tokio)

Characteristics:

• Pros: Automatic load balancing, efficient CPU utilization, handles CPU-bound and I/O-bound tasks
• Cons: More complex scheduler, potential for work-stealing overhead
• Used by: Go runtime, Tokio (Rust), Rayon (Rust)

4. Actor Model (Erlang, Akka)

Characteristics:

• Pros: No shared state (no locks!), fault isolation (one actor crash doesn't kill others), natural distributed model
• Cons: Message passing overhead, potential mailbox overflow, harder to reason about data flow
• Used by: Erlang/Elixir (BEAM VM), Akka (JVM), Orleans (.NET)

Model Comparison

Why Redis is Single-Threaded (Yet Fast)

When to Use Each Model

Real Numbers: Event Loop Performance

// Benchmark: HTTP echo server, 100K concurrent connections
// Setup: Linux 5.15, 16-core Xeon

Node.js (single thread):        45K req/s (1 core saturated)
Node.js (cluster mode, 16):     520K req/s (linear scaling)

Nginx (multi-reactor, 16):      680K req/s, 8ms p99 latency
Go (work-stealing, 16):         620K req/s, 10ms p99 latency

Erlang (actor model, 16):       450K req/s, 15ms p99 latency
  (Higher latency due to message passing overhead)

// CPU-bound task impact (1ms compute per request)
Node.js (single thread):        900 req/s (blocked!)
Go (goroutines):                14K req/s (utilizes all cores)
Rust (Tokio):                   15K req/s

// Context: Why multi-reactor beats work-stealing for I/O
Multi-reactor: Minimal contention (per-core event loops)
Work-stealing: Some contention (shared run queue, stealing overhead)
Result: ~10% faster for pure I/O workloads

2.12 Signals and Interrupts

Hardware Interrupts vs Software Interrupts

Interrupt Handling Flow

Interrupt Coalescing (Adaptive Batching)

Problem: At high packet rates (1M+ pps), interrupt overhead dominates CPU time.

// 1M packets/sec = 1 IRQ per μs
Packet 1 → IRQ → Process (3 μs)
Packet 2 → IRQ → Process (3 μs)
Packet 3 → IRQ → Process (3 μs)
...

Total CPU time: 3M μs = 3 CPU cores

// Batch 50 packets per IRQ
Packets 1-50 → Wait 50 μs → IRQ → Process batch (10 μs)
Packets 51-100 → Wait 50 μs → IRQ → Process batch (10 μs)
...

Total IRQs: 20K/sec (vs 1M/sec)
Total CPU time: 200K μs = 0.2 CPU cores

Interrupt Coalescing Tuning

# View current settings
ethtool -c eth0

# Tune interrupt coalescing
# rx-usecs: Delay before IRQ (μs)
# rx-frames: Max packets before IRQ (even if delay not elapsed)

# Low latency (default)
ethtool -C eth0 rx-usecs 0 rx-frames 1
# Result: IRQ per packet, 1-3 μs latency, high CPU

# Balanced (good for most workloads)
ethtool -C eth0 rx-usecs 50 rx-frames 32
# Result: ~50 μs latency, moderate CPU

# High throughput (batch processing)
ethtool -C eth0 rx-usecs 200 rx-frames 128
# Result: ~200 μs latency, low CPU, high throughput

# Adaptive (driver adjusts based on load)
ethtool -C eth0 adaptive-rx on
# Result: Low latency under low load, batching under high load

NAPI: Interrupt to Polling Transition

NAPI (New API) automatically switches between interrupt mode (low load) and polling mode (high load).

IRQ Affinity for CPU Pinning

Goal: Pin IRQ handlers to specific CPUs to improve cache locality.

# Find IRQ number for eth0
cat /proc/interrupts | grep eth0
# Output: 42: eth0-TxRx-0

# Pin IRQ 42 to CPU 0 (bitmask: 0x01)
echo 1 > /proc/irq/42/smp_affinity

# Pin IRQ 43 to CPU 1 (bitmask: 0x02)
echo 2 > /proc/irq/43/smp_affinity

# Pin IRQ 44 to CPUs 2-3 (bitmask: 0x0C = 1100 binary)
echo c > /proc/irq/44/smp_affinity

# Automated: Use irqbalance daemon
systemctl start irqbalance
# irqbalance dynamically distributes IRQs across CPUs

How Interrupts Affect Tail Latency

Scenario: Latency-sensitive application (target p99 < 5ms)

Mitigating Interrupt Latency

Linux Signals

Asynchronous notifications sent to processes. Common signals:

Signal Handling Best Practices

// C++ signal handler example
#include <signal.h>
#include <unistd.h>
#include <atomic>

std::atomic<bool> shutdown_requested(false);

void signal_handler(int signum) {
    // IMPORTANT: Signal handlers must be async-signal-safe!
    // Safe: write(), _exit(), atomic ops
    // Unsafe: printf(), malloc(), most library functions

    if (signum == SIGTERM || signum == SIGINT) {
        shutdown_requested.store(true);
        // Write to pipe to wake event loop
        write(shutdown_pipe, "X", 1);
    }
}

int main() {
    // Install signal handlers
    signal(SIGTERM, signal_handler);
    signal(SIGINT, signal_handler);

    // Event loop
    while (!shutdown_requested.load()) {
        // Process requests
        handle_request();
    }

    // Graceful shutdown
    close_connections();
    flush_logs();
    return 0;
}

Real Numbers: Interrupt Performance

// Benchmark: 10GbE network, UDP flood test
// Setup: Linux 5.15, Intel X710 NIC, 16-core Xeon

No tuning (interrupt per packet):
  Packet rate:              600K pps
  CPU usage (si%):          65% (IRQ overhead)
  Application throughput:   Low (starved by IRQs)

Interrupt coalescing (rx-usecs=50):
  Packet rate:              1.2M pps (+100%)
  CPU usage (si%):          15% (batched IRQs)
  Latency impact:           +50 μs (acceptable)

NAPI polling (netdev_budget=600):
  Packet rate:              2.5M pps (+4x)
  CPU usage (si%):          8% (mostly polling)
  Latency:                  ~20 μs

IRQ affinity (pin to cores 0-7):
  Application p99 latency:  4ms (vs 12ms without pinning)
  Context switches:         -80% (no IRQ preemption)

DPDK (kernel bypass):
  Packet rate:              14M pps (+23x)
  CPU usage:                100% (dedicated polling)
  Latency:                  ~3 μs (no interrupts)