Chapter 25: Event Streaming

Event streaming platforms like Apache Kafka provide distributed, durable, ordered logs for event-driven architectures. Unlike traditional message queues, event streams enable replay, time-travel, and multi-consumer patterns at massive scale.

1. Event Streaming vs Message Queuing

Event streaming and message queuing solve different problems. Queues (RabbitMQ, SQS) deliver messages to consumers and delete them. Streams (Kafka, Pulsar) retain events as an immutable log, enabling replay and time-based queries.

Key Differences

Aspect	Message Queue (RabbitMQ, SQS)	Event Stream (Kafka, Pulsar)
Message Lifetime	Deleted after consumption (ack)	Retained by time/size (days/weeks)
Replay	No replay (message gone after ack)	Rewind to any offset, reprocess
Consumer Model	Competing consumers (one gets msg)	Consumer groups + broadcast (multiple)
Ordering	Queue-level or no guarantee	Per-partition ordering (strict)
Use Case	Task distribution, RPC, work queues	Event sourcing, analytics, CDC, logs
Throughput	Lower (ack overhead, per-message)	Higher (batching, sequential disk I/O)
Offset Management	Broker tracks acks	Consumer tracks offset (flexible)

2. Kafka Architecture

Kafka is a distributed commit log. Producers append to topics, which are split into partitions. Brokers store partitions, and consumers read at their own pace. ZooKeeper (legacy) or KRaft (new) manage cluster metadata.

Core Components

Component	Description	Details
Broker	Kafka server node	Stores partitions, serves produce/fetch requests. Cluster = multiple brokers.
Topic	Logical event stream	Category/feed name (e.g., "orders", "clicks"). Divided into partitions.
Partition	Ordered, immutable log	Unit of parallelism. Messages appended sequentially. One leader, N replicas.
Segment	Physical log file	Partition split into segments (1GB or 7 days). Old segments deleted/compacted.
Offset	Message position	Sequential ID within partition (0, 1, 2...). Immutable.
ZooKeeper	Cluster coordination (legacy)	Leader election, broker membership. Being replaced by KRaft.
KRaft	Kafka Raft (new)	Built-in consensus (no ZooKeeper dependency). GA in Kafka 3.3+.

3. Producers

Producers publish events to topics. Key decisions: partitioning strategy (which partition?), acknowledgment level (durability vs latency), batching (throughput vs delay), and idempotence (exactly-once).

Partitioning Strategies

Strategy	How It Works	Use Case	Trade-offs
Key-based	hash(key) % partitions	Ordering per key (e.g., user_id)	Skew if keys unbalanced. Default for keyed messages.
Round-robin	Cycle through partitions	No key, max throughput	No ordering guarantees. Balances load.
Sticky	Batch to one partition, then switch	High throughput (default no-key)	Better batching than round-robin. Kafka 2.4+.
Custom	Implement Partitioner interface	Domain-specific (geo, tenant)	Complexity. Must avoid hotspots.

Acknowledgment Levels (acks)

acks	Durability	Latency	Risk
0	None (fire-and-forget)	Lowest	Data loss if broker fails before write. Use for metrics, logs.
1	Leader acknowledges	Medium	Data loss if leader fails before replication. Default.
all (-1)	Leader + all ISR replicas ack	Highest	No loss (if min.insync.replicas=2+). Use for critical data.

Producer Configuration Essentials

Setting	Default	Purpose	Tuning
batch.size	16KB	Batch size in bytes	Increase (64KB-1MB) for throughput. Adds latency.
linger.ms	0	Wait time to fill batch	Set 5-100ms for throughput. Adds delay.
compression.type	none	Compress batches	Use snappy (fast) or lz4. Saves network/storage.
enable.idempotence	true (3.0+)	Deduplication	Prevents duplicates on retry. Requires acks=all.
max.in.flight	5	Unacked requests	Lower (1) for strict ordering. Higher for throughput.

4. Consumers & Consumer Groups

Consumers read from topics at their own pace. Consumer groups enable horizontal scaling: each partition assigned to one consumer in the group. Multiple groups can read the same topic independently.

Consumer Group Mechanics

Concept	Description	Details
Consumer Group	Logical subscriber	group.id identifies group. Partitions split among group members.
Partition Assignment	One partition → one consumer	No two consumers in same group read same partition. Max parallelism = partition count.
Rebalancing	Reassign partitions	Triggered by consumer join/leave/failure. Stops consumption during rebalance.
Group Coordinator	Broker managing group	One broker per group. Handles heartbeats, rebalances, offset commits.
Offset	Consumer position	Stored in __consumer_offsets topic. Auto-commit or manual.

Rebalancing Strategies

Strategy	How It Works	Pros	Cons
Eager (Range, RoundRobin)	Stop all, reassign all	Simple. Default.	Stop-the-world. All consumers pause.
Cooperative (Incremental)	Revoke only moved partitions	Minimal downtime. Consumers keep unaffected partitions.	Two-phase (revoke, assign). Kafka 2.4+.
Sticky	Minimize partition movement	Preserves assignments. Reduces cache misses.	More complex. Works with eager or cooperative.

Offset Management

Mode	How It Works	Use Case	Risk
Auto-commit	Commit every N seconds	Simple. enable.auto.commit=true.	At-least-once (duplicates on crash). At-most-once if commit before process.
Manual (sync)	commitSync() after processing	At-least-once. Process then commit.	Duplicates on crash. Blocks thread.
Manual (async)	commitAsync() after processing	At-least-once. Non-blocking.	Duplicates on crash. May commit out of order.
Transactional	Commit offset in transaction	Exactly-once (with transactional producer).	Requires isolation.level=read_committed. See section 6.

Consumer Lag

Consumer lag = (log end offset) - (consumer offset). High lag = consumer falling behind. Monitor via kafka-consumer-groups.sh, Burrow, or metrics (kafka.consumer:type=consumer-fetch-manager-metrics,partition=*).

5. Partitions Deep Dive

Partitions are the unit of parallelism, ordering, and replication. Messages within a partition are strictly ordered. Across partitions, no ordering guarantee. Choose partition count carefully (hard to increase later).

Ordering Guarantees

Scope	Guarantee	Caveat
Within Partition	Total order	Messages appended sequentially. Consumer reads in order (offset 0, 1, 2...).
Across Partitions	No order	Producer may write P0, P1, P0. Consumer sees interleaved.
Producer Retries	Idempotent preserves order	With enable.idempotence=true + max.in.flight ≤ 5. Else retries may reorder.

Partition Count Selection

Factor	Guidance	Notes
Throughput	partitions = target_throughput / partition_throughput	Each partition: ~10-30MB/s producer, ~20-50MB/s consumer (disk-bound).
Consumer Parallelism	Max consumers = partition count	More partitions = more parallel consumers. Diminishing returns after ~100.
Latency	More partitions = longer rebalance	Rebalance time grows with partition count. 1000s of partitions = minutes.
Memory	Each partition = buffer in producer/consumer	Producer: batch.size per partition. Consumer: fetch.min.bytes.
File Handles	Each partition = open files (log, index)	Broker may hit ulimit with 10K+ partitions.

Replication & In-Sync Replicas (ISR)

Concept	Description	Details
Replication Factor	Number of replicas	RF=3 means 1 leader + 2 followers. Tolerates RF-1 failures.
Leader	Partition owner	All produce/fetch requests go to leader. One leader per partition.
Follower	Replica	Fetches from leader. Passive (no client requests). Becomes leader on failure.
ISR (In-Sync Replicas)	Caught-up replicas	Leader + followers within replica.lag.time.max.ms (default 10s). acks=all waits for ISR.
OSR (Out-of-Sync)	Lagging replicas	Removed from ISR. Cannot become leader (data loss risk). Must catch up to rejoin.
min.insync.replicas	Min ISR for acks=all	Topic config. Set 2 for RF=3 (leader + 1 follower). Producer fails if ISR < min.

6. Exactly-Once Semantics

Exactly-once semantics (EOS) ensure each message is processed exactly once, even with failures. Requires idempotent producer (deduplication) + transactional API (atomic read-process-write).

Delivery Guarantees

Guarantee	How	Trade-offs
At-most-once	Commit offset, then process	Data loss on crash. Fast. Rarely desired.
At-least-once	Process, then commit offset	Duplicates on crash. Default. Requires idempotent downstream.
Exactly-once	Transactional read-process-write	No duplicates. Higher latency, complexity. Kafka-to-Kafka only.

Exactly-Once Components

Component	Purpose	Configuration
Idempotent Producer	Deduplicates retries	enable.idempotence=true (default 3.0+). Assigns PID + sequence number.
Transactional Producer	Atomic multi-partition writes	transactional.id (unique per producer). Enables transactions.
Transactional Consumer	Read only committed messages	isolation.level=read_committed. Skips aborted transactions.
Transaction Coordinator	Manages transaction state	Broker component. Writes to __transaction_state topic.

Read-Process-Write Pattern

// Exactly-once Kafka-to-Kafka processing
Properties props = new Properties();
props.put("enable.idempotence", true);
props.put("transactional.id", "my-app-txn-id-0");

producer.initTransactions();

while (true) {
    ConsumerRecords records = consumer.poll(100);
    producer.beginTransaction();

    for (ConsumerRecord record : records) {
        // Process
        ProducerRecord output = transform(record);
        producer.send(output);
    }

    // Commit offset in transaction
    producer.sendOffsetsToTransaction(
        consumer.position(), consumer.groupMetadata()
    );

    producer.commitTransaction(); // Atomic
}

Exactly-Once Limitations

Limitation	Details
Kafka-to-Kafka Only	EOS guarantees only for read-from-Kafka, write-to-Kafka. External systems (DB, API) not covered.
Performance	Adds latency (2-phase commit). ~50% throughput vs non-transactional.
Zombie Fencing	transactional.id must be unique. Two producers with same ID = one fenced (aborted).
Consumer Lag	read_committed consumers lag by transaction duration (wait for commit/abort marker).

7. Kafka Streams & ksqlDB

Kafka Streams is a Java library for stream processing (no separate cluster). ksqlDB is a SQL interface for streams. Both enable stateful transformations, joins, windowing on Kafka topics.

Kafka Streams Concepts

Concept	Description	Use Case
KStream	Unbounded event stream	Append-only log. Each record = new event. (clicks, orders)
KTable	Changelog stream (latest per key)	Upsert semantics. Key = primary key. (user profile, inventory)
GlobalKTable	Replicated KTable (all partitions)	Small reference data (lookup, broadcast join). Higher memory.
State Store	Local RocksDB storage	Stateful operations (aggregations, joins). Backed by changelog topic.
Windowing	Group by time window	Tumbling (non-overlapping), hopping (overlapping), session (gap-based).

Stream Processing Operations

Operation	Type	Description	Stateful?
map, filter	Stateless	Transform/filter each record	No
groupBy, aggregate	Stateful	Group by key, compute aggregates (count, sum)	Yes (state store)
join (KStream-KStream)	Stateful	Join events within time window	Yes (both sides buffered)
join (KStream-KTable)	Stateful	Enrich stream with table (latest)	Yes (table cached)
join (KTable-KTable)	Stateful	Join two tables (foreign key)	Yes (both cached)

ksqlDB

ksqlDB adds SQL interface for Kafka Streams. Run persistent queries (continuous) or pull queries (point-in-time). Backed by Kafka Streams under the hood.

-- Create stream
CREATE STREAM orders (order_id INT, user_id INT, amount DOUBLE)
  WITH (KAFKA_TOPIC='orders', VALUE_FORMAT='JSON');

-- Windowed aggregation
CREATE TABLE sales_per_minute AS
  SELECT user_id, COUNT(*) as order_count, SUM(amount) as total
  FROM orders
  WINDOW TUMBLING (SIZE 1 MINUTE)
  GROUP BY user_id
  EMIT CHANGES;

-- Stream-table join
CREATE STREAM enriched_orders AS
  SELECT o.order_id, o.amount, u.name, u.email
  FROM orders o
  JOIN users u ON o.user_id = u.user_id
  EMIT CHANGES;

8. Schema Registry

Schema Registry (Confluent component) stores schemas for Kafka messages. Producers/consumers validate against schemas. Enables evolution (add fields) without breaking consumers.

Schema Registry Features

Feature	Description	Details
Schema Storage	Centralized schema repo	REST API. Schemas stored in _schemas topic. Versioned per subject.
Schema Formats	Avro, Protobuf, JSON Schema	Avro most common (compact, fast). Protobuf for cross-language. JSON Schema for flexibility.
Schema ID	Embedded in message	4-byte schema ID prefix. Consumer fetches schema from registry (cached locally).
Compatibility Checks	Enforce evolution rules	Registry validates new schemas against compatibility mode. Rejects incompatible changes.

Compatibility Modes

Mode	Rule	Use Case
BACKWARD	New schema reads old data	Default. Add optional fields, remove fields. Consumers upgrade first.
FORWARD	Old schema reads new data	Remove optional fields, add fields. Producers upgrade first.
FULL	Backward + forward	Only add/remove optional fields. Any upgrade order.
NONE	No checks	Disable validation. Risky (runtime errors).
BACKWARD_TRANSITIVE	Compatible with ALL previous	Check against all versions, not just latest.

Schema Evolution Example (Avro)

// Version 1
{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "order_id", "type": "int"},
    {"name": "amount", "type": "double"}
  ]
}

// Version 2 (BACKWARD compatible)
{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "order_id", "type": "int"},
    {"name": "amount", "type": "double"},
    {"name": "currency", "type": "string", "default": "USD"} // Optional with default
  ]
}

9. Kafka Connect

Kafka Connect is a framework for streaming data between Kafka and external systems (databases, object stores, APIs). Source connectors import data into Kafka. Sink connectors export from Kafka.

Kafka Connect Components

Component	Description	Details
Connector	Logical job	Config (source/sink, topic, transform). Manages tasks.
Task	Worker thread	Executes data movement. Multiple tasks for parallelism.
Worker	Connect runtime	Standalone (single process) or distributed (cluster). REST API for management.
Converter	Serialization format	JSON, Avro, Protobuf. Decouples connector from format.
Transform	Single-message modification	SMT (Single Message Transform). Mask, rename, route. No aggregations.

Popular Connectors

Connector	Type	Use Case	Notes
Debezium (CDC)	Source	Capture DB changes (MySQL, Postgres, etc)	Reads transaction log. Low-latency, no polling.
JDBC Sink	Sink	Write to relational DB	Upsert, insert, delete. Auto-create tables.
S3 Sink	Sink	Archive to object storage	Parquet, JSON, Avro. Partitioning by time/field.
Elasticsearch Sink	Sink	Index for search	Bulk indexing. Schema mapping.
HTTP Source/Sink	Both	REST API integration	Poll API (source), POST (sink).

Change Data Capture (CDC) with Debezium

Debezium captures row-level changes (insert, update, delete) from database transaction logs. Publishes to Kafka with before/after snapshots. Enables event sourcing, cache invalidation, analytics.

// Debezium MySQL connector config
{
  "name": "mysql-cdc",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "mysql",
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "secret",
    "database.server.id": "184054",
    "database.server.name": "prod",
    "table.include.list": "inventory.orders",
    "schema.history.internal.kafka.topic": "schema-changes.inventory"
  }
}

// Output to topic: prod.inventory.orders
// Message: {"before": {...}, "after": {...}, "op": "u"}

10. Operational Concerns

Running Kafka in production requires attention to retention, compaction, partition management, monitoring, and disaster recovery.

Retention Policies

Policy	How It Works	Configuration
Time-based	Delete segments older than N	retention.ms (default 7 days). Per topic or global.
Size-based	Delete when partition exceeds size	retention.bytes. Use for bounded storage.
Log Compaction	Keep only latest per key	cleanup.policy=compact. For changelog topics (KTable).

Log Compaction

Compaction retains the latest value for each key, deleting older versions. Guarantees each key appears at least once. Useful for changelog topics (user profiles, inventory). Background process merges segments.

Aspect	Details
Use Case	KTable backing, CDC snapshots, configuration topics
Retention	Keeps latest per key forever (unless deleted via tombstone: null value)
Process	Cleaner thread merges segments, rewrites without old values
Config	cleanup.policy=compact, min.cleanable.dirty.ratio (default 0.5)

Partition Reassignment

Move partitions between brokers for load balancing, broker decommission, or rack awareness. Use kafka-reassign-partitions.sh or Cruise Control (LinkedIn's auto-rebalancing tool).

Monitoring Metrics

Metric	What It Measures	Alert On
Under-Replicated Partitions	Partitions with ISR < RF	kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions > 0
Consumer Lag	Messages behind	kafka.consumer:type=consumer-fetch-manager-metrics,partition=*,topic=*,attribute=records-lag-max
Request Latency	Produce/fetch time	kafka.network:type=RequestMetrics,name=TotalTimeMs,request={Produce|Fetch} p99 > 100ms
Active Controllers	Controller count	kafka.controller:type=KafkaController,name=ActiveControllerCount != 1
Offline Partitions	Partitions with no leader	kafka.controller:type=KafkaController,name=OfflinePartitionsCount > 0

Disaster Recovery

Strategy	How	RTO/RPO
Cross-Cluster Replication	MirrorMaker 2.0 (Kafka Connect)	RPO: seconds (async). RTO: minutes (DNS/client switch).
Backup to S3	S3 Sink Connector	RPO: hours (batch). RTO: hours (restore + rebuild).
Active-Active	Bidirectional MirrorMaker + conflict resolution	RPO: zero. RTO: zero (if app can handle both clusters).

11. Alternatives to Kafka

Kafka dominates event streaming, but alternatives exist for specific use cases: Pulsar (multi-tenancy, tiered storage), Redpanda (no JVM, low latency), managed services (Kinesis, EventHubs).

Kafka vs Alternatives

Platform	Strengths	Weaknesses	When to Use
Kafka	Mature, ecosystem, throughput, Kafka Streams	JVM (GC pauses), ZooKeeper (legacy), ops complexity	Default choice. Event sourcing, CDC, analytics.
Pulsar	Multi-tenancy, tiered storage (S3), geo-replication, functions	Smaller ecosystem, fewer managed options	Multi-tenant SaaS, infinite retention (tiered), geo-active.
Redpanda	C++ (no JVM), low latency, Kafka API compatible, simpler ops	Newer (less mature), smaller community	Performance-critical, want Kafka API without JVM.
AWS Kinesis	Fully managed, auto-scaling, AWS integration	Proprietary API, shard limits (1MB/s write), higher cost	AWS-native, small scale, don't want ops.
Azure EventHubs	Fully managed, Kafka API support, Azure integration	Kafka API limited (no admin ops), cost	Azure-native, want managed Kafka-like.
NATS JetStream	Simple, low latency, lightweight, K8s-native	Simpler feature set, smaller ecosystem	Microservices messaging, edge, IoT.

Pulsar Architecture Highlights

Feature	How It Differs from Kafka
Layered Architecture	Serving layer (brokers) + storage layer (BookKeeper). Decoupled. Kafka = monolithic broker.
Tiered Storage	Auto-offload old segments to S3/GCS. Infinite retention without broker disk. Kafka requires add-on.
Multi-Tenancy	Namespaces, quotas, isolation built-in. Kafka = single tenant or manual ACLs.
Geo-Replication	Built-in async replication across clusters. Kafka requires MirrorMaker.

When to Choose What

Scenario	Best Choice
Event sourcing, CDC, stream processing	Kafka (mature, Kafka Streams, ecosystem)
Multi-tenant SaaS (isolated tenants)	Pulsar (namespaces, quotas)
Low-latency trading, gaming	Redpanda (C++, no GC) or Pulsar
Infinite retention (years of data)	Pulsar (tiered storage) or Kafka + S3 sink
AWS-only, small scale, minimal ops	Kinesis
Microservices pub/sub (simple)	NATS JetStream or RabbitMQ

---

Benchmarks & Production Patterns

Kafka Performance Benchmarks

Configuration	Hardware	Throughput (msgs/sec)	Throughput (MB/sec)	P99 Latency (acks=all)
3-broker cluster	m5.2xlarge, EBS gp3	800K msgs/sec	800 MB/sec	15ms produce
Single partition write	m5.xlarge	~10K msgs/sec	~10 MB/sec	5-10ms
Single partition read	m5.xlarge	~30K msgs/sec	~30 MB/sec	1-5ms fetch

Production Scale Examples

Company	Volume	Infrastructure	Use Case
LinkedIn	7 trillion messages/day	100+ Kafka clusters, 100K+ topics. Largest cluster: 3000 brokers.	Activity streams, metrics, logs, CDC, messaging
Netflix	700 billion events/day	Multiple Kafka clusters globally. Keystone pipeline (custom).	Playback events, A/B test logging, personalization signals, operational metrics
Uber	1+ trillion messages/day	Multi-region Kafka clusters, custom uReplicator (MirrorMaker fork)	Surge pricing, ETA calculation, fraud detection, dispatch events

Partition Count Formula

Rule of thumb: partitions = max(throughput / partition_throughput, consumer_count)

Scenario	Target Throughput	Partition Throughput	Partition Count
Single partition write limit	-	~10 MB/sec write	Disk-bound (sequential I/O)
Single partition read limit	-	~30 MB/sec read	Network-bound (page cache)
Example: 100 MB/sec write	100 MB/sec	10 MB/sec per partition	100 MB / 10 MB = 10 partitions
Example: 20 consumers	-	-	Min 20 partitions (1 per consumer)

Kafka vs Pulsar vs Redpanda Concrete Comparison

Metric	Kafka	Pulsar	Redpanda
P99 Latency	~15ms (acks=all)	~7-10ms (2x lower tail latency)	~1-5ms (10x lower P99, no JVM GC)
Storage Architecture	Monolithic (broker = storage)	Tiered (BookKeeper + S3 offload)	Monolithic (C++ Raft, local disk)
Operational Complexity	High (ZooKeeper/KRaft, JVM tuning)	Higher (BookKeeper + ZooKeeper + brokers)	Low (single binary, no ZooKeeper)
Ecosystem	Largest (10x more connectors, libraries)	Smaller (Pulsar Functions, built-in geo-replication)	Smallest (Kafka API compatible, growing)
Multi-Tenancy	Manual (ACLs, quotas)	Native (namespaces, tenant isolation)	Manual (via Kafka API compatibility)
Infinite Retention	Requires S3 sink connector	Built-in (tiered storage to S3/GCS)	Tiered storage (beta, v22.2+)

Real-World Architecture: Netflix Keystone Pipeline

Use case: 700 billion events/day for playback telemetry, personalization, and A/B testing.

Component	Technology	Details
Ingestion	Kafka (custom build)	Multi-region clusters. 100K+ partitions across topics.
Stream Processing	Flink (not Kafka Streams)	Real-time aggregations, windowing. Chosen for richer feature set than Kafka Streams.
Storage	S3 + Parquet	Cold storage for analytics. S3 sink from Kafka via custom connector.
Real-Time	ElasticSearch + Druid	Kafka → ES for search, Kafka → Druid for time-series analytics.
Monitoring	Consumer lag, under-replicated partitions	Custom tooling. Alert on lag > 1M messages or ISR < min.insync.replicas.

Summary

Event streaming platforms enable distributed, durable, replayable event logs at massive scale. Kafka remains the dominant choice, with partitioning for parallelism, consumer groups for load distribution, and exactly-once semantics for correctness. Pulsar offers multi-tenancy and tiered storage; Redpanda removes JVM overhead. Choose based on scale, latency, ops complexity, and ecosystem maturity.