Chapter 71: ML Systems at Scale

Feature Stores

A feature store is a centralized repository for storing, managing, and serving ML features. It solves the problem of feature inconsistency between training and serving (training/serving skew).

Online vs Offline Feature Stores

Feature Store Architecture

graph TB
    subgraph "Feature Creation"
        Raw[Raw Data Sources]
        Stream[Streaming Data]
        Batch[Batch Data]
    end

    subgraph "Feature Store"
        Registry[Feature Registry
Metadata, Lineage, Schema]
        Offline[Offline Store
S3/BigQuery]
        Online[Online Store
Redis/DynamoDB]
    end

    subgraph "Consumers"
        Training[Model Training]
        Serving[Model Serving]
        Notebook[Notebooks/Analysis]
    end

    Raw --> Stream
    Raw --> Batch
    Stream --> |Real-time pipeline| Online
    Batch --> |Batch pipeline| Offline
    Offline --> |Materialize hot features| Online

    Registry -.-> Offline
    Registry -.-> Online

    Offline --> Training
    Offline --> Notebook
    Online --> Serving

Popular Feature Store Solutions

Feature Store Benefits

• Consistency: Same feature computation for training and serving
• Reusability: Share features across models and teams
• Point-in-time correctness: Prevent data leakage in training
• Monitoring: Track feature drift and data quality
• Discovery: Catalog of available features

Model Serving

Model serving is the process of making trained models available for inference. The architecture differs dramatically based on latency requirements.

Batch vs Real-Time Serving

Real-Time Model Serving Architecture

graph LR
    Client[Client Application]
    LB[Load Balancer]

    subgraph "Model Serving Cluster"
        MS1[Model Server 1
GPU/CPU]
        MS2[Model Server 2
GPU/CPU]
        MS3[Model Server 3
GPU/CPU]
    end

    subgraph "Supporting Services"
        FS[Feature Store
Online]
        Cache[Prediction Cache
Redis]
        Monitor[Monitoring
Prometheus]
    end

    Client --> LB
    LB --> MS1
    LB --> MS2
    LB --> MS3

    MS1 --> FS
    MS2 --> FS
    MS3 --> FS

    MS1 --> Cache
    MS2 --> Cache
    MS3 --> Cache

    MS1 --> Monitor
    MS2 --> Monitor
    MS3 --> Monitor

Model Serving Frameworks

Optimization Techniques

• Model quantization: Reduce precision (FP32 → INT8) for 4x speedup
• Model pruning: Remove low-importance weights
• Knowledge distillation: Train small model to mimic large model
• Request batching: Process multiple requests together
• Caching: Cache predictions for common inputs
• Model compilation: TensorRT, TVM for hardware-specific optimization

Model Training Pipeline

A robust training pipeline automates the process from raw data to deployed model, ensuring reproducibility and enabling continuous improvement.

Training Pipeline Stages

graph TD
    Data[1. Data Collection
Raw logs, events, databases]
    Clean[2. Data Cleaning
Deduplication, outlier removal]
    Features[3. Feature Engineering
Transformations, aggregations]
    Split[4. Data Splitting
Train/Val/Test 70/15/15]
    Train[5. Model Training
Hyperparameter tuning]
    Eval[6. Model Evaluation
Metrics, validation]
    Register[7. Model Registry
Version, metadata]
    Deploy[8. Deployment
Canary, A/B test]
    Monitor[9. Monitoring
Drift, performance]

    Data --> Clean
    Clean --> Features
    Features --> Split
    Split --> Train
    Train --> Eval
    Eval --> |Metrics meet threshold| Register
    Eval --> |Metrics poor| Train
    Register --> Deploy
    Deploy --> Monitor
    Monitor --> |Drift detected| Data

Key Components

Model Evaluation Metrics

A/B Testing for ML Models

A/B testing is the gold standard for evaluating model improvements in production. Offline metrics don't capture real user behavior and system interactions.

A/B Test Framework

graph TB
    User[User Request]
    Assign[Experiment Assignment
User ID hash]

    subgraph "Control Group 50%"
        ModelA[Model A
Current Production]
        LogA[Log: user_id, model_a, metrics]
    end

    subgraph "Treatment Group 50%"
        ModelB[Model B
New Candidate]
        LogB[Log: user_id, model_b, metrics]
    end

    Analysis[Statistical Analysis
T-test, Confidence Interval]
    Decision[Launch Decision
Rollout or Rollback]

    User --> Assign
    Assign --> |50%| ModelA
    Assign --> |50%| ModelB
    ModelA --> LogA
    ModelB --> LogB
    LogA --> Analysis
    LogB --> Analysis
    Analysis --> Decision

Key Concepts

Guardrail Metrics

Primary metric is what you optimize for, but guardrail metrics ensure you don't break the system.

Common Pitfalls

• Peeking: Checking results early and stopping test biases results
• Multiple testing: Running 20 tests increases false positive rate
• Novelty effect: Users initially like new UI, then revert
• Carryover effect: Treatment affects control group (network effects)
• Sample ratio mismatch: 50/50 split ends up 48/52 (logging bug)

Feedback Loops

Feedback loops occur when a model's predictions influence the data used to retrain it. This can create positive feedback (reinforcing) or negative feedback (correcting).

Positive Feedback Loops (Dangerous)

graph TD
    Model[Model predicts
Item A popular]
    Show[System shows
Item A to users]
    Click[Users click
Item A more]
    Data[Training data:
Item A has high CTR]
    Retrain[Retrain model
with new data]

    Model --> Show
    Show --> Click
    Click --> Data
    Data --> Retrain
    Retrain --> Model

    style Model fill:#ffcccc
    style Retrain fill:#ffcccc

Real-World Examples

Breaking Feedback Loops

• Exploration: ε-greedy (random 10% of time), Thompson sampling
• Counterfactual logging: Log what model would have predicted for all items
• Propensity scoring: Weight training samples by probability they were shown
• Manual curation: Human review of edge cases
• Adversarial testing: Red team tries to game the system

MLOps (ML Operations)

MLOps is the practice of deploying, monitoring, and maintaining ML models in production. It's the ML equivalent of DevOps.

MLOps Maturity Levels

Model Versioning

Monitoring ML Models

graph TB
    subgraph "Input Monitoring"
        InputDist[Input Distribution
Feature drift detection]
        DataQuality[Data Quality
Missing values, outliers]
    end

    subgraph "Model Monitoring"
        Predictions[Prediction Distribution
Output drift]
        Confidence[Confidence Scores
Uncertainty estimation]
    end

    subgraph "Performance Monitoring"
        Online[Online Metrics
CTR, conversion, revenue]
        Offline[Offline Metrics
Precision, recall, RMSE]
        Latency[Latency
p50, p99, p999]
    end

    subgraph "Alerts & Actions"
        Alert[Alert on Drift
Slack, PagerDuty]
        Retrain[Trigger Retraining
Pipeline]
        Rollback[Rollback Model
Previous version]
    end

    InputDist --> Alert
    DataQuality --> Alert
    Predictions --> Alert
    Online --> Alert
    Offline --> Alert

    Alert --> Retrain
    Alert --> Rollback

Model Drift

Retraining Triggers

• Scheduled: Daily/weekly retraining regardless of performance
• Performance-based: Retrain when accuracy drops below threshold
• Drift-based: Retrain when input distribution shifts (PSI > 0.2)
• Data volume: Retrain when N new labeled samples accumulated
• Manual: Data scientist triggers after investigation

Production ML Challenges & Solutions

The gap between training models and running them in production is vast. Here are the real-world challenges you'll face at scale.

Training-Serving Skew

The #1 production ML issue. Features are computed differently in training (batch, historical) versus serving (real-time, live).

Model Staleness

ML models degrade over time as the world changes. How often should you retrain?

GPU Serving Economics

GPUs are expensive but can be cheaper than CPUs at high throughput. Here's the math:

Canary Deployment for Models

Never deploy a model to 100% of traffic immediately. Roll out gradually and watch metrics.

// Canary deployment strategy
1. Deploy new model (v2) to 5% of traffic
2. Run for 24 hours, compare metrics:
   - Click-through rate (CTR)
   - Revenue per user
   - Latency (p50, p99)
   - Error rate

3. Decision criteria:
   - If CTR drops >2%: AUTO-ROLLBACK to v1
   - If revenue down >5%: AUTO-ROLLBACK
   - If p99 latency >100ms: AUTO-ROLLBACK
   - If all metrics stable/better: Ramp to 25%, then 50%, then 100%

4. Full rollout takes 3-5 days for safety

Shadow Mode

Before canary, run new model in shadow mode: it gets 100% of traffic but predictions are logged, not served.

• Production model (v1): Serves actual predictions to users
• Shadow model (v2): Receives same inputs, logs predictions, no user impact
• Comparison: Analyze shadow predictions offline: "If we'd used v2, would metrics improve?"
• Zero risk: If v2 is broken, no user sees it