Chapter 72: Real-Time Analytics

1. Batch Layer (Accuracy)

• Purpose: Compute accurate views from complete historical data
• Storage: Immutable, append-only master dataset (HDFS, S3)
• Processing: Apache Spark, Hadoop MapReduce
• Frequency: Hourly or daily batch jobs
• Output: Pre-computed batch views
• Latency: Hours (fresh data delayed)

2. Speed Layer (Latency)

• Purpose: Provide real-time views for recent data
• Processing: Apache Flink, Storm, Spark Streaming
• State: In-memory or fast storage (Redis, Cassandra)
• Approximation: May sacrifice accuracy for speed
• Output: Incremental real-time views
• Latency: Seconds to subseconds

3. Serving Layer (Queries)

• Purpose: Merge batch and real-time views for queries
• Storage: Druid, Cassandra, HBase
• Query logic: batch_view(old data) + speed_view(recent data)
• Interface: REST API, SQL interface

Requirement: Show pageview counts by URL for any time range.

Batch Layer

• Runs daily: aggregate all pageviews from past 90 days
• Output: {"/home": 1M views, "/product": 500K views, ...}
• Accurate but data is up to 24 hours old

Speed Layer

• Processes pageview events in real-time
• Maintains counts for last 24 hours
• Output: {"/home": +5K views today, "/product": +2K views today}

Serving Layer

• Query: "Show pageviews for /home in last 7 days"
• Response: batch_view("/home", 6 days ago) + speed_view("/home", today)
• Result: Accurate historical + up-to-the-second recent data

1. Everything is a Stream

• Historical data: replay from durable event log (Kafka)
• Real-time data: process as it arrives
• No distinction between batch and streaming

2. Reprocessing for Changes

• Need to change logic? Start new stream processor
• Replay all historical events from beginning
• Write to new serving table
• Cutover when reprocessing catches up

3. Single Codebase

• One stream processing job handles all data
• No separate batch/speed layer logic
• Easier to maintain and test

Scenario: Count user actions per day. Need to change counting logic.

Without Kappa (Painful)

1. Update batch job code → wait 24 hours for next run
2. Historical data reprocessed, but slow
3. Real-time and batch out of sync during transition

With Kappa (Smooth)

1. Deploy new stream processor (v2 logic)
2. Replay Kafka from day 1 at high speed (parallel consumers)
3. Write to new table: user_activity_v2
4. When caught up, redirect queries to v2
5. Shutdown old processor

Use Lambda When

• Historical data too large to replay (petabytes)
• Batch processing tools already in place (Spark jobs)
• Different algorithms needed for batch vs real-time
• Regulatory requirements for exact accuracy

Use Kappa When

• Historical data fits in Kafka retention (days to months)
• Frequent logic changes (need fast reprocessing)
• Team prefers simpler architecture
• Stream processing can handle full data volume

Hybrid Approach

• Use Kappa for recent data (last 30 days in Kafka)
• Use batch for deep historical analysis (years in S3)
• Combine results at query time

Goal: Update materialized view without recomputing everything.

Example: COUNT

• Old value: 1000 pageviews for /home
• New events: 5 pageviews for /home
• Incremental update: 1000 + 5 = 1005

Example: SUM

• Old value: $10,000 total revenue
• New events: $500 in new orders
• Incremental update: $10,000 + $500 = $10,500

Challenge: AVG

• Cannot incrementally update average directly
• Solution: Store SUM and COUNT, compute AVG = SUM / COUNT
• Incrementally update SUM and COUNT separately

PostgreSQL Materialized Views

CREATE MATERIALIZED VIEW daily_sales AS
SELECT date, product_id, SUM(amount) AS total_sales
FROM orders
GROUP BY date, product_id;

-- Refresh manually
REFRESH MATERIALIZED VIEW daily_sales;

-- Refresh concurrently (non-blocking)
REFRESH MATERIALIZED VIEW CONCURRENTLY daily_sales;

Apache Flink / Kafka Streams (Real-Time)

// Continuously update materialized view from stream
stream
  .groupBy(event -> event.productId)
  .aggregate(new SumAggregator())
  .to("materialized_view_topic");

Druid (Rollup)

• Automatically rolls up data at ingestion time
• Stores pre-aggregated metrics (SUM, COUNT, MIN, MAX)
• Queries run on rolled-up data, not raw events

Event Time

• Definition: When the event actually occurred (timestamp in event)
• Pros: Accurate, handles late arrivals, replayable
• Cons: Complex (watermarks, late data handling)
• Example: User clicked button at 10:05:23 AM

Processing Time

• Definition: When the event was processed by the system
• Pros: Simple, no late data
• Cons: Not reproducible, skewed by delays
• Example: Event processed at 10:07:15 AM (2 min delay)

// Real-time analytics: Count pageviews per URL in 5-minute windows
DataStream<PageView> stream = env.addSource(new KafkaSource(...));

stream
  .keyBy(view -> view.url)
  .window(TumblingEventTimeWindows.of(Time.minutes(5)))
  .allowedLateness(Time.minutes(1))  // Handle late data
  .aggregate(new CountAggregator())
  .addSink(new CassandraSink(...));  // Write to serving DB

Key Features

• Event time: Use timestamp from event, not processing time
• Watermarks: Track progress of event time
• Allowed lateness: Accept events up to 1 min late, update results
• Exactly-once: Flink guarantees no duplicates or lost data

1. Slice

Select a single value from one dimension.

• Example: "Show sales for Region = 'North America'"
• Result: Reduces 3D cube to 2D (Product × Time)

2. Dice

Select a sub-cube by filtering multiple dimensions.

• Example: "Show sales for Product IN ('A', 'B') AND Time IN (Q3, Q4)"
• Result: Smaller 3D cube

3. Drill-Down

Navigate from summary to detail.

• Example: Year → Quarter → Month → Day
• Use case: User clicks "Q4" to see monthly breakdown

4. Roll-Up

Aggregate from detail to summary.

• Example: Day → Month → Quarter → Year
• Use case: Summarize daily sales into monthly totals

5. Pivot (Rotate)

Swap dimensions to change perspective.

• Example: Change from "Products by Region" to "Regions by Product"
• Use case: Excel pivot tables

Problem: Pre-computing all combinations explodes exponentially.

Example

• 10 dimensions, each with 100 values
• Possible combinations: 100^10 = 10^20 cells
• Even with sparse data, storage becomes massive

Solutions

• Partial cubes: Pre-compute only common queries
• Aggregation hierarchies: Store only summary levels
• Sparse storage: Don't store empty cells
• Sampling: Approximate results for large cubes

1. Caching Hot Metrics

• Store frequently queried metrics in Redis
• Update cache from stream processor
• Serve dashboard queries from cache (sub-ms latency)
• Example: "Current active users" updated every second

2. Pre-Aggregation

• Stream processor computes aggregates (hourly, daily)
• Dashboard queries pre-aggregated data, not raw events
• Reduces query time from seconds to milliseconds

3. Downsampling

• Reduce resolution for old data
• Example: Last hour: 1-second granularity → Last day: 1-minute → Last month: 1-hour
• Saves storage and speeds up queries

4. Query Result Caching

• Cache dashboard query results for 5-30 seconds
• Many users viewing same dashboard hit cache
• Trade-off: slight staleness for massive speedup

Architecture

• Data source: Prometheus, InfluxDB, Druid
• Queries: PromQL, SQL, custom DSL
• Refresh interval: 5s to 1min (configurable)
• Alerting: Trigger alerts on threshold breach

Best Practices

• Limit panels per dashboard (5-10) to avoid query overload
• Use variables for filtering (time range, region, service)
• Pre-aggregate data for historical views (rollups)
• Optimize queries: avoid SELECT * , use indexed columns

Requirements

• Show active rides, driver availability, demand heatmaps
• Update every 1-5 seconds
• Support 10,000+ simultaneous viewers (ops team, execs)

Solution

• Stream processing: Flink aggregates ride events by city, time window
• Serving layer: Pre-aggregated data in Pinot (real-time OLAP)
• Caching: Redis caches common queries (city-level metrics)
• Push updates: WebSocket pushes new data to connected clients
• Result: Sub-second latency for 10K+ concurrent users