Chapter 57: Design Amazon/E-commerce Platform

1. Requirements

Functional Requirements

Feature Description Priority
Product Catalog Browse millions of products across hierarchical categories with rich attributes P0
Search & Discovery Full-text search, autocomplete, faceted filtering, personalized recommendations P0
Shopping Cart Add/remove items, update quantities, persist across sessions P0
Checkout Multi-step checkout with address, payment, order confirmation P0
Order Management Order placement, tracking, history, cancellations, returns P0
Inventory Management Real-time stock tracking across warehouses, oversell prevention P0
Payment Processing Multiple payment methods, fraud detection, secure transactions P0
Fulfillment Warehouse allocation, shipping provider integration, delivery tracking P0
Reviews & Ratings User reviews, ratings, verified purchases, review helpfulness P1
Wishlist Save items for later, share wishlists, price alerts P1

Non-Functional Requirements

Requirement Target Strategy
Scale 300M active users, 100M daily visitors, 1B+ products Horizontal scaling, sharding, caching
Black Friday Load 10x normal traffic, 100K orders/minute peak Auto-scaling, queue-based order processing, CDN
Availability 99.99% uptime (52min downtime/year) Multi-region, circuit breakers, graceful degradation
Latency Product page: <200ms, Search: <300ms, Checkout: <500ms CDN, Redis caching, read replicas
Consistency Strong for orders/payments/inventory, eventual for catalog/reviews Different consistency models per domain
Data Durability Zero order/payment data loss Multi-region replication, point-in-time recovery
Interview Tip: Clarify read vs write ratios early. E-commerce is heavily read-dominated (browse:buy ratio ~100:1). Product pages get millions of views but relatively few purchases. This drastically affects architecture decisions (aggressive caching, CDN, read replicas).

Traffic Estimates

Metric Normal Load Black Friday Peak
Daily Active Users 100M 250M
Product Page Views/sec 500K 5M
Search Queries/sec 100K 1M
Cart Operations/sec 50K 500K
Orders/sec 1K 10K
Inventory Updates/sec 5K 50K

2. High-Level Design

graph TB subgraph "Client Layer" Web[Web Browser] Mobile[Mobile App] API[Partner APIs] end subgraph "CDN & Load Balancing" CDN[CloudFront CDN] LB[Load Balancer] end subgraph "API Gateway" Gateway[API Gateway
Auth, Rate Limiting] end subgraph "Application Services" Catalog[Catalog Service] Search[Search Service] Cart[Cart Service] Order[Order Service] Inventory[Inventory Service] Payment[Payment Service] Fulfillment[Fulfillment Service] Notification[Notification Service] User[User Service] Review[Review Service] end subgraph "Data Stores" CatalogDB[(Catalog DB
Postgres)] OrderDB[(Order DB
Postgres)] UserDB[(User DB
Postgres)] InventoryDB[(Inventory DB
Postgres)] ES[Elasticsearch
Search Index] Redis[(Redis
Cache & Cart)] end subgraph "Message Queue" Queue[Kafka/SQS] end subgraph "External Services" PaymentGW[Payment Gateway
Stripe/PayPal] ShippingAPI[Shipping APIs
UPS/FedEx] EmailSMS[Email/SMS
Twilio/SendGrid] end subgraph "Storage" S3[S3
Product Images/Docs] end Web --> CDN Mobile --> CDN API --> LB CDN --> LB LB --> Gateway Gateway --> Catalog Gateway --> Search Gateway --> Cart Gateway --> Order Gateway --> User Gateway --> Review Catalog --> CatalogDB Catalog --> Redis Catalog --> S3 Catalog --> ES Search --> ES Search --> Redis Cart --> Redis Order --> OrderDB Order --> Queue Order --> Redis User --> UserDB User --> Redis Inventory --> InventoryDB Inventory --> Redis Inventory --> Queue Payment --> Queue Payment --> PaymentGW Fulfillment --> Queue Fulfillment --> ShippingAPI Notification --> Queue Notification --> EmailSMS Review --> CatalogDB Review --> Redis Queue --> Payment Queue --> Fulfillment Queue --> Inventory Queue --> Notification

Service Responsibilities

Service Responsibilities Data Store Scaling Strategy
Catalog Service Product CRUD, categories, attributes, images Postgres + Redis cache Read replicas, CDN for images
Search Service Full-text search, autocomplete, facets, ranking Elasticsearch cluster Horizontal sharding by product_id
Cart Service Add/remove items, quantity updates, price calculation Redis (primary), DB backup Redis cluster, session affinity
Order Service Order creation, state transitions, history Postgres + Kafka events Shard by user_id
Inventory Service Stock tracking, reservation, warehouse allocation Postgres + Redis cache Shard by product_id, optimistic locking
Payment Service Payment processing, fraud detection, refunds Postgres (transactional) Idempotency, retry with backoff
Fulfillment Service Shipping provider selection, tracking, delivery Postgres + external APIs Async processing via queue
Notification Service Email/SMS for orders, shipping, promotions Queue + delivery logs Queue workers, rate limiting
Interview Tip: Emphasize event-driven architecture for order processing. Order placement triggers events consumed by inventory, payment, fulfillment, and notification services. This decouples services and allows async processing of non-critical path operations.

3. Product Catalog

Catalog Schema Design

Table Key Columns Purpose
products product_id, title, brand, base_price, category_id, created_at Core product information
categories category_id, parent_id, name, level, path Hierarchical category tree
product_attributes product_id, attribute_key, attribute_value, data_type Variable attributes by product type (EAV pattern)
product_images image_id, product_id, url, display_order, is_primary Product images (CDN URLs)
product_variants variant_id, product_id, sku, size, color, price, stock Product variations (size/color combinations)
sellers seller_id, name, rating, fulfillment_method Third-party seller information
product_sellers product_id, seller_id, price, condition, shipping_cost Multiple sellers per product

Category Hierarchy Example

category_id parent_id name level path
1 NULL Electronics 0 /1/
10 1 Computers 1 /1/10/
100 10 Laptops 2 /1/10/100/
101 10 Desktops 2 /1/10/101/
11 1 Mobile Phones 1 /1/11/

Product Attributes (EAV Pattern)

Product Type Sample Attributes Why Variable Schema
Laptop screen_size, ram, processor, gpu, storage_type, battery_life Tech specs vary widely
Shirt fabric, fit, collar_type, sleeve_length, care_instructions Apparel has different attributes
Book author, publisher, isbn, page_count, publication_date, language Completely different schema
Refrigerator capacity, energy_rating, door_type, defrost_type, warranty Appliance-specific attributes

Caching Strategy

Cache Layer What's Cached TTL Invalidation
CDN (CloudFront) Product images, static assets 30 days Version-based URLs
Redis (Product Details) Full product object with attributes 1 hour Write-through on updates
Redis (Category Tree) Complete category hierarchy 24 hours Manual invalidation (rare changes)
Application Cache Hot products (top 1000) 5 minutes Time-based expiration
Interview Tip: Discuss EAV (Entity-Attribute-Value) vs JSON column vs dedicated tables trade-offs. EAV is flexible but query performance suffers. JSON columns (Postgres JSONB) offer flexibility + indexing. Dedicated tables are fastest but require schema changes. For Amazon scale, hybrid approach: core attributes in columns, extended attributes in JSONB.

5. Shopping Cart

Cart Architecture Comparison

Approach Storage Pros Cons
Session-Based Browser cookies/localStorage No server load, works for anonymous users Lost on device switch, no backup, security risk
Database-Backed Postgres table (user_id, product_id, quantity, added_at) Persistent, multi-device, queryable High DB load, slower, overkill for abandoned carts
Redis (Recommended) Redis hash per user (cart:{user_id} → {product_id: quantity}) Fast, persistent, scalable, multi-device Requires Redis cluster, eventual backup to DB
Hybrid Redis primary, DB backup (daily snapshot) Performance + durability Complexity of sync

Cart Schema (Redis)

Key Pattern Data Structure Fields TTL
cart:{user_id} Hash {product_id: quantity, variant_id: quantity} 30 days (sliding window on update)
cart_meta:{user_id} Hash last_updated, item_count, total_price 30 days
cart_session:{session_id} Hash For anonymous users, merged on login 7 days

Cart Operations

Operation Redis Command Additional Logic
Add Item HSET cart:{user_id} {product_id} {quantity} Check inventory availability, update cart_meta
Update Quantity HINCRBY cart:{user_id} {product_id} {delta} Validate quantity > 0, check stock limits
Remove Item HDEL cart:{user_id} {product_id} Update cart_meta totals
Get Cart HGETALL cart:{user_id} Hydrate with product details (price, title, image) from cache
Clear Cart DEL cart:{user_id} After successful order placement
Merge Carts (Login) HGETALL cart_session:{sid}, HSET cart:{uid} Union of session + user cart, handle duplicates

Cart Merge Strategy (Anonymous → Logged In)

Scenario Strategy Example
No conflict Union of both carts Session: [A, B], User: [C] → Result: [A, B, C]
Same product Sum quantities (with max limit check) Session: [A:2], User: [A:3] → Result: [A:5]
Same variant Keep higher quantity or sum (configurable) Session: [A-red:1], User: [A-red:2] → Result: [A-red:3]
Out of stock Remove item, notify user Session: [A:5] but stock=3 → Result: [], show alert

Cart Challenges & Solutions

Challenge Problem Solution
Price Changes Product price changed since cart add Always fetch fresh price at checkout, show diff if changed
Stock Depletion Item was available but now out of stock Validate inventory at checkout, show "unavailable" message
Abandoned Carts 90%+ carts never convert to orders TTL (30 days), reminder emails, retargeting ads
Concurrent Updates User updates cart from multiple devices Redis atomic operations, last-write-wins for quantity
Cart Size Limits Prevent abuse (1000+ items in cart) Max 100 unique items, max 999 quantity per item
Interview Tip: Discuss cart persistence trade-offs. Redis offers best performance but requires backup strategy. DB-only is safe but slow. Hybrid with Redis + periodic DB snapshots is production-grade. Emphasize TTL for abandoned carts to manage memory.

6. Inventory Management

Inventory Schema

Table Key Columns Purpose
inventory product_id, warehouse_id, available_stock, reserved_stock, total_stock, version Real-time stock tracking per warehouse
warehouses warehouse_id, location, region, capacity Warehouse metadata
inventory_reservations reservation_id, product_id, user_id, quantity, expires_at, status Temporary stock holds during checkout
inventory_logs log_id, product_id, warehouse_id, change_type, quantity, timestamp Audit trail for all stock changes

Oversell Prevention Strategies

Approach Mechanism Pros Cons
Pessimistic Locking SELECT FOR UPDATE, lock row during checkout Guaranteed accuracy, no oversell Low throughput, lock contention, deadlocks
Optimistic Locking Version column, UPDATE WHERE version=old_version High throughput, no locks Retry logic needed, can fail under high contention
Redis Atomic Decrement DECRBY stock:{product_id} {quantity}, check if result >= 0 Very fast, atomic, scales horizontally Requires cache-DB sync, eventual consistency
Inventory Reservation Reserve stock for 10min, release if not purchased Balances accuracy + availability, good UX Requires expiration job, temporary stock reduction

Recommended: Optimistic Locking + Reservation

Step Action SQL/Pseudocode
1. Check Availability Read available stock SELECT available_stock, version FROM inventory WHERE product_id=? AND warehouse_id=?
2. Reserve Stock Atomic decrement with version check UPDATE inventory SET available_stock=available_stock-?, reserved_stock=reserved_stock+?, version=version+1 WHERE product_id=? AND version=? AND available_stock >= ?
3. Check Update If 0 rows updated, retry or fail if (rows_affected == 0) { retry or return "out of stock" }
4. Create Reservation Insert reservation record INSERT INTO inventory_reservations (product_id, user_id, quantity, expires_at) VALUES (?, ?, ?, NOW()+10min)
5. Confirm/Release On order: move reserved→sold, else: reserved→available UPDATE inventory SET reserved_stock=reserved_stock-?, total_stock=total_stock-? (on purchase) OR available_stock=available_stock+?, reserved_stock=reserved_stock-? (on expiry)

Warehouse Allocation Algorithm

Criteria Weight Logic
Proximity 50% Nearest warehouse to shipping address (minimize delivery time)
Stock Availability 30% Fulfill entire order from one warehouse if possible
Warehouse Load 10% Balance load across warehouses (avoid overwhelming single facility)
Shipping Cost 10% Prefer warehouses with cheaper shipping rates

Inventory Sync (DB → Cache)

Strategy Mechanism Consistency
Write-Through Update DB + Redis in same transaction Strong, but slower writes
Write-Behind (Async) Update Redis immediately, DB via queue Eventual, risk of data loss
CDC (Change Data Capture) Postgres WAL → Debezium → Kafka → Redis consumer Eventual (~100ms lag), reliable
Periodic Poll Redis TTL 5min, cache miss triggers DB read Eventual, simple but higher DB load
Interview Tip: Emphasize reservation system for high-demand items. During Black Friday flash sales, pessimistic locking would crush throughput. Optimistic locking + short-lived reservations (10min) balances accuracy with performance. Failed reservations get retried or user is notified "out of stock".

7. Order Processing

Order State Machine

stateDiagram-v2 [*] --> PENDING: Order Placed PENDING --> PAYMENT_PROCESSING: Initiate Payment PAYMENT_PROCESSING --> CONFIRMED: Payment Success PAYMENT_PROCESSING --> PAYMENT_FAILED: Payment Declined PAYMENT_FAILED --> PENDING: Retry Payment PAYMENT_FAILED --> CANCELLED: Max Retries/User Cancel CONFIRMED --> PROCESSING: Inventory Reserved PROCESSING --> SHIPPED: Package Dispatched SHIPPED --> IN_TRANSIT: Carrier Picked Up IN_TRANSIT --> OUT_FOR_DELIVERY: At Local Facility OUT_FOR_DELIVERY --> DELIVERED: Customer Received CONFIRMED --> CANCELLED: User Cancels (before ship) PROCESSING --> CANCELLED: User Cancels (before ship) DELIVERED --> RETURN_REQUESTED: User Requests Return RETURN_REQUESTED --> RETURNED: Item Received Back DELIVERED --> [*]: Complete RETURNED --> [*]: Refunded CANCELLED --> [*]: Cancelled

Order Schema

Table Key Columns Purpose
orders order_id, user_id, status, total_amount, shipping_address_id, created_at, updated_at Order header
order_items item_id, order_id, product_id, variant_id, quantity, unit_price, subtotal Line items
order_events event_id, order_id, event_type, timestamp, metadata (JSON) Audit log for state transitions
payments payment_id, order_id, amount, method, status, transaction_id, created_at Payment records
shipments shipment_id, order_id, carrier, tracking_number, shipped_at, delivered_at Shipping details

Event-Driven Order Flow

Event Producer Consumers Actions
OrderPlaced Order Service Inventory, Payment, Notification Reserve inventory, initiate payment, send confirmation email
PaymentSucceeded Payment Service Order, Inventory, Fulfillment, Notification Update order status to CONFIRMED, allocate warehouse, send success email
PaymentFailed Payment Service Order, Inventory, Notification Release reserved inventory, notify user to retry
OrderShipped Fulfillment Service Order, Notification Update order status to SHIPPED, send tracking email
OrderDelivered Fulfillment Service Order, Review, Notification Mark DELIVERED, prompt for review, send delivery confirmation
OrderCancelled Order Service Payment, Inventory, Notification Refund payment, release inventory, send cancellation email

Order State Validation Rules

Transition Allowed From Validation
PENDING → CONFIRMED PENDING Payment succeeded, inventory reserved
CONFIRMED → CANCELLED CONFIRMED, PROCESSING Not yet shipped, refund payment, release inventory
PROCESSING → SHIPPED PROCESSING Package handed to carrier, tracking number assigned
SHIPPED → DELIVERED SHIPPED, IN_TRANSIT, OUT_FOR_DELIVERY Carrier confirms delivery, signature/photo proof
DELIVERED → RETURN_REQUESTED DELIVERED Within return window (30 days), item returnable

Idempotency for Order Creation

Problem Solution Implementation
Double Submit Idempotency key (client-generated UUID) Store key in orders table (unique constraint), reject duplicate requests
Retry After Timeout Return existing order if key already processed SELECT order_id FROM orders WHERE idempotency_key=?
Partial Failure Saga pattern with compensating transactions If payment fails after inventory reserve, release inventory via OrderCancelled event
Interview Tip: Emphasize event-driven architecture for order processing. Synchronous calls (Order → Payment → Inventory → Fulfillment) create tight coupling and timeout risks. Async events via Kafka allow services to process independently, retry on failure, and scale separately. Use saga pattern for distributed transactions.

8. Checkout & Payment

Checkout Flow

Step Action Validation Failure Handling
1. Cart Review Display cart items, prices, total Recalculate prices (may have changed), check inventory Show price changes, remove OOS items
2. Shipping Address User enters/selects address Validate format, check serviceable area Suggest corrections, show unserviceable message
3. Shipping Method Standard, Express, Overnight options Calculate cost based on address + weight Show only available options
4. Payment Method Credit card, PayPal, gift card, etc. Validate card number (Luhn), expiry, CVV Show validation errors
5. Review & Confirm Summary of order details Final inventory check If OOS, prevent order placement
6. Reserve Inventory Hold stock for 10 minutes Optimistic locking to prevent oversell If reserve fails, show "item unavailable"
7. Process Payment Charge via payment gateway 3DS authentication, fraud check Retry (network), show error (declined), release inventory
8. Create Order Insert order record with idempotency key Ensure no duplicate orders Return existing order if key matches
9. Confirm Inventory Move reserved → sold Deduct from available stock Rollback order if inventory update fails
10. Order Confirmation Display order number, send email Emit OrderPlaced event N/A (order already created)

Payment Gateway Integration

Payment Method Gateway Flow Considerations
Credit/Debit Card Stripe, Braintree Tokenize card → Charge API → 3DS auth if needed PCI compliance (never store raw card numbers), fraud detection
PayPal PayPal API Redirect to PayPal → User approves → Callback to confirm Handle redirect delays, verify callback authenticity
Apple Pay / Google Pay Stripe, Braintree Mobile SDK → Payment token → Charge API Device-specific, requires merchant account approval
Buy Now Pay Later (BNPL) Affirm, Klarna Redirect → Credit check → Installment plan → Callback Async approval, handle declined credit applications
Gift Card Internal system Validate balance → Deduct amount → Apply to order Combine with other methods if balance insufficient

Payment Failure Handling

Failure Type Cause Retry Strategy User Action
Declined Insufficient funds, fraud block, wrong CVV No auto-retry User updates payment method or contacts bank
Network Timeout Gateway unreachable, slow response Exponential backoff (3 retries) Show "processing" spinner, then error if all fail
Gateway Error (5xx) Gateway internal issue Retry with exponential backoff Show "try again later" if persists
3DS Authentication Failed User cancelled, wrong OTP No auto-retry User re-attempts checkout
Duplicate Charge (rare) Retry after timeout but first charge succeeded Idempotency key prevents double charge None (system handles automatically)

Checkout Optimization Strategies

Optimization Technique Impact
Reduce Steps Single-page checkout instead of multi-step wizard 15-20% higher conversion rate
Guest Checkout Allow purchase without account creation 30% of users prefer guest checkout
Auto-Fill Browser autofill, saved addresses for logged-in users Faster checkout, fewer errors
Real-Time Validation Inline validation for address/card fields Reduce form submission errors by 40%
Progress Indicator Show step 1 of 3, 2 of 3, etc. Reduce abandonment (users know time commitment)
Save Payment Methods Tokenize and store for future use One-click reorder for repeat customers
Interview Tip: Discuss the "reserve inventory → charge payment → confirm order" sequence. If you charge first and inventory is unavailable, you must refund (bad UX). If you reserve inventory first, you prevent overselling. Reservations expire (10min) to avoid locking stock indefinitely. Emphasize idempotency for payment retries.

9. Fulfillment

Warehouse Selection Algorithm

Factor Calculation Weight
Distance to Customer Haversine formula (lat/lon) or zip code proximity 50%
Stock Availability Can fulfill entire order from one warehouse? 30%
Warehouse Capacity Current load vs max capacity (avoid overload) 10%
Shipping Cost Estimated cost from warehouse to customer 10%

Fulfillment Process

Stage Action System Update
1. Order Assignment Allocate order to warehouse(s) based on algorithm Create pick list for warehouse staff
2. Picking Warehouse worker collects items from shelves Scan each item, update pick status
3. Quality Check Verify correct items, no damage Flag issues for resolution
4. Packing Pack items, add invoice, seal box Calculate weight, dimensions
5. Label Generation Generate shipping label with tracking number Call carrier API to create shipment
6. Handoff to Carrier Carrier picks up package from warehouse Update order status to SHIPPED, emit OrderShipped event
7. Tracking Updates Carrier provides status updates (in transit, delivered) Webhook from carrier → update shipment table
8. Delivery Confirmation Customer receives package, signature/photo Update order status to DELIVERED, emit OrderDelivered event

Shipping Provider Integration

Carrier API Capabilities Webhook Events
UPS Rate calculation, label generation, tracking, address validation In transit, out for delivery, delivered, exception (delay)
FedEx Rate calculation, label generation, tracking, pickup scheduling Picked up, in transit, delivered, returned to sender
USPS Rate calculation, label generation (limited tracking) Accepted, in transit, delivered (less granular)
DHL International shipping, customs documentation, tracking Customs cleared, in transit, delivered

Tracking & Notifications

Tracking Event Customer Notification Internal Action
Order Confirmed Email: "Your order has been confirmed" Log event, start fulfillment process
Shipped Email + SMS: "Your order has shipped" + tracking link Update order status, emit event
In Transit Push notification: "Your order is on the way" Update tracking info
Out for Delivery SMS: "Your order will arrive today" Update ETA
Delivered Email + Push: "Your order has been delivered" Mark complete, prompt for review
Delivery Exception Email: "Delivery delayed due to [reason]" Alert support team, investigate

Returns Processing

Step Action System Update
1. Return Request User selects item(s), reason, refund method Create return request, generate RMA number
2. Return Label Generate prepaid shipping label for customer Email label, track return shipment
3. Item Received Warehouse receives returned item, inspects condition Update return status to RECEIVED
4. Quality Check Verify item matches return, check for damage Approve or reject return
5. Refund Process refund to original payment method Call payment gateway refund API, update order
6. Restock Return item to inventory if resellable Increment available_stock, log inventory change
Interview Tip: Discuss split shipments for multi-item orders. If items are in different warehouses, shipping separately may be faster than consolidating. Trade-off: customer experience (prefers single package) vs speed + cost. For high-value customers, consider consolidation; for speed-focused orders (Prime), split shipments are acceptable.

10. Scalability & Performance

Read-Heavy Optimization (Browsing)

Technique Implementation Benefit
CDN for Product Pages Cache entire product HTML at edge (CloudFront) Offload 90% of product page traffic from origin
Redis for Product Data Cache product objects (title, price, images, attributes) Reduce DB reads by 95%, <5ms latency
Read Replicas 5-10 Postgres read replicas per write master Distribute read queries, handle 100K QPS
Elasticsearch for Search Horizontally scale search cluster (50+ nodes) Handle millions of search queries/sec
Image CDN Store images in S3, serve via CloudFront Fast global delivery, reduce origin load

Write-Spike Optimization (Black Friday)

Challenge Solution Mechanism
Order Flood Queue-based order processing API accepts order → Kafka queue → Workers process async
Payment Gateway Limits Rate limiting, circuit breaker, multi-gateway Fallback to secondary gateway if primary is slow
Inventory Contention Optimistic locking, Redis atomic ops Handle high concurrency without locks
Database Write Load Batch writes, write-behind cache Buffer writes in memory, flush in batches
Notification Storm Rate-limited email/SMS queues Avoid overwhelming notification providers

Database Sharding

Table Shard Key Rationale
products product_id (hash) Evenly distribute product data, no hot shards
orders user_id (hash) User's order history on same shard (efficient queries)
inventory product_id (hash) Co-locate inventory with product data
users user_id (hash) User data + orders on same shard
reviews product_id (hash) Reviews for a product on same shard as product

Caching Strategy Summary

Layer Technology Hit Rate Target TTL
CDN (Edge) CloudFront 85% for static content 30 days (images), 1 hour (pages)
Application Cache Redis Cluster 95% for product data 1 hour (products), 5 min (inventory)
Database Query Cache Postgres (built-in) 50% for repeated queries Automatic invalidation
Search Cache Elasticsearch (node-level) 70% for popular queries 5 minutes

Auto-Scaling Configuration

Service Metric Scale Up Threshold Scale Down Threshold
API Servers CPU utilization >70% for 3 min <30% for 10 min
Search Service Query latency P95 >500ms P95 <200ms for 15 min
Order Workers Queue depth >1000 messages <100 messages for 10 min
Redis Cluster Memory usage >80% <50% for 30 min
Database Read Replicas Read QPS >5K per replica <1K per replica for 20 min

Graceful Degradation Strategies

Scenario Degradation User Impact
Search Service Down Fall back to category browse No search bar, can still navigate categories
Review Service Down Show product without reviews Reviews section hidden, purchase still possible
Recommendation Service Down Show top-sellers instead of personalized Less relevant recommendations, no blocker
Payment Gateway Timeout Retry with backoff, queue order if persists Slower checkout, eventual success
Inventory Service Down Allow add to cart, validate at checkout May fail at checkout if OOS, but browsing works

Monitoring & Alerting

Metric Alert Threshold Action
Order Success Rate <99% Investigate payment/inventory failures
Search Latency P95 >500ms Scale Elasticsearch, check query performance
Inventory Oversells >10 per hour Review locking mechanism, increase reservation time
Payment Gateway Errors >5% failure rate Switch to backup gateway, contact provider
Cart Abandonment Rate >75% Review checkout UX, check for bugs
Database Replication Lag >5 seconds Scale read replicas, optimize slow queries
Interview Tip: Black Friday is the ultimate stress test. Emphasize pre-scaling (add capacity before peak), queue-based order processing (handle 10x normal load), and graceful degradation (reviews/recommendations can fail, but checkout must work). Discuss load testing with 10x traffic weeks before the event.
Key Takeaways: E-commerce is heavily read-optimized (browse:buy ~100:1) with write spikes during sales. Use CDN + Redis + read replicas for product pages, queue-based async processing for orders, optimistic locking for inventory, event-driven architecture for cross-service coordination, and comprehensive monitoring for rapid incident response. Database sharding by product_id (catalog) and user_id (orders) enables horizontal scaling. Emphasize idempotency for payments and reservations for inventory to balance accuracy with performance.

Inventory Reservation Deep Dive

Step-by-Step Flow with Sequence Diagram

sequenceDiagram participant User participant API as Order API participant Inv as Inventory Service participant DB as Inventory DB participant Pay as Payment Service participant BG as Background Job User->>API: Click "Buy Now" API->>API: Create order (status=PENDING) Note over API,Inv: Step 2: Reserve Inventory API->>Inv: Reserve stock for product X, qty=1 Inv->>DB: UPDATE stock SET reserved=reserved+1
WHERE product_id=X AND (available-reserved) >= 1 alt UPDATE affected 0 rows DB-->>Inv: 0 rows affected (out of stock) Inv-->>API: Reservation failed API->>API: Cancel order API-->>User: Error: Item out of stock else UPDATE successful DB-->>Inv: 1 row affected Inv->>DB: INSERT reservation (TTL=15min, reservation_id) Inv-->>API: Reservation created (reservation_id) API->>API: Store reservation_id on order Note over API,Pay: Step 5: Charge Payment API->>Pay: Charge card (idempotent with order_id) alt Payment succeeds Pay-->>API: Payment success API->>Inv: Confirm reservation (convert to committed) Inv->>DB: UPDATE stock SET available=available-1, reserved=reserved-1 API->>API: Order status=CONFIRMED API-->>User: Order successful! else Payment fails Pay-->>API: Payment declined API->>Inv: Release reservation Inv->>DB: UPDATE stock SET reserved=reserved-1 API-->>User: Payment failed, stock released end end Note over BG: Step 8: Background Job (every 1min) loop Every 1 minute BG->>DB: SELECT * FROM reservations WHERE created_at < NOW()-15min BG->>DB: UPDATE stock SET reserved=reserved-1 WHERE reservation_id IN (...) BG->>DB: DELETE FROM reservations WHERE expired end

Detailed Steps

Step Action SQL / Pseudocode
1. Order Creation User clicks "Buy" → API creates order with status=PENDING INSERT INTO orders (order_id, user_id, status) VALUES (?, ?, 'PENDING')
2. Reserve Stock (Atomic) Inventory service performs atomic check-and-set UPDATE inventory SET reserved=reserved+1 WHERE product_id=? AND (available - reserved) >= ? AND warehouse_id=?
3. Check Result If UPDATE affected 0 rows → item out of stock → return error, cancel order if (rows_affected == 0) { cancel_order(); return "Out of stock"; }
4. Create Reservation If success → reservation created with TTL=15min, reservation_id stored on order INSERT INTO inventory_reservations (reservation_id, product_id, order_id, quantity, expires_at) VALUES (?, ?, ?, ?, NOW()+INTERVAL 15 MINUTE)
5. Payment Processing Payment service charges card (idempotent with order_id as key) payment_gateway.charge(amount, card_token, idempotency_key=order_id)
6. Payment Success If payment succeeds → confirm order, convert reservation to committed UPDATE inventory SET available=available-1, reserved=reserved-1 WHERE product_id=?
UPDATE orders SET status='CONFIRMED'
7. Payment Failure If payment fails → release reservation UPDATE inventory SET reserved=reserved-1 WHERE product_id=?
DELETE FROM inventory_reservations WHERE reservation_id=?
8. Background Cleanup Every 1min, scan reservations older than 15min → release them (handles abandoned carts, payment timeouts) SELECT reservation_id, product_id, quantity FROM inventory_reservations WHERE expires_at < NOW()
UPDATE inventory SET reserved=reserved-quantity WHERE product_id IN (...)
DELETE FROM inventory_reservations WHERE expires_at < NOW()

Race Condition: Two Users Buy Last Item Simultaneously

Scenario: Product has available=1, reserved=0. At time T, User A and User B both click "Buy Now" for the same item.

Time User A Thread User B Thread DB State (available, reserved)
T+0ms Initial state (1, 0)
T+10ms UPDATE reserved=reserved+1 WHERE available-reserved >= 1 UPDATE reserved=reserved+1 WHERE available-reserved >= 1 (1, 0) → executing
T+11ms Row-level lock acquired, WHERE check passes (1-0=1 >= 1) Waiting for lock... (1, 1) User A wins
T+12ms Lock released, UPDATE returns 1 row affected Lock acquired, WHERE check fails (1-1=0 < 1) (1, 1)
T+13ms Reservation success → proceed to payment UPDATE returns 0 rows → Out of stock (1, 1)

Solution: The atomic UPDATE ... WHERE (available - reserved) >= quantity is the gate. Database row-level locks ensure only one transaction succeeds. The WHERE clause check happens INSIDE the lock, so the check-and-set is atomic. User B's transaction sees the updated reserved count and fails the WHERE condition.

WRONG Approach (Has Race Condition)

# BAD: Check then update (race condition!)
stock = SELECT available, reserved FROM inventory WHERE product_id=?
if (stock.available - stock.reserved > 0) {
  # ⚠️ Another thread can check here too!
  UPDATE inventory SET reserved=reserved+1 WHERE product_id=?
  # Both threads pass the check → oversell!
}

Problem: The check and update are separate operations. Between the check and the update, another thread can also pass the check, leading to oversell.

Oversell Prevention at Scale: Flash Sale (10K Orders/Sec)

Problem: Pessimistic locking (SELECT FOR UPDATE) = 10K lock acquisitions/sec on a single row. This creates a bottleneck. Every order waits for the lock.

Solution: Pre-Allocate Inventory Buckets

Approach Mechanism Example
Bucket Strategy Divide total inventory into N buckets. Route requests round-robin across buckets. 10K items → 10 buckets × 1000 items each. Bucket 0: rows 0-999, Bucket 1: rows 1000-1999, etc.
Load Distribution Hash user_id or session_id to assign bucket: bucket_id = hash(user_id) % 10 User A → bucket 3, User B → bucket 7. Locks are distributed across 10 rows instead of 1.
Contention Reduction Each bucket handles ~1K orders/sec instead of 10K on single row 10x reduction in lock contention. 10 parallel UPDATE operations instead of 1 serialized queue.
Fallback If assigned bucket is empty, try next bucket (wrap-around) Bucket 3 depleted → try bucket 4, 5, ... until stock found or all exhausted.

Bucket Implementation

# Schema: inventory_buckets table
CREATE TABLE inventory_buckets (
  product_id INT,
  bucket_id INT,
  available INT,
  reserved INT,
  PRIMARY KEY (product_id, bucket_id)
);

# Pre-populate for flash sale
INSERT INTO inventory_buckets VALUES
  (12345, 0, 1000, 0),
  (12345, 1, 1000, 0),
  ...
  (12345, 9, 1000, 0);

# Reserve from assigned bucket
bucket_id = hash(user_id) % 10;
UPDATE inventory_buckets
SET reserved=reserved+1
WHERE product_id=? AND bucket_id=? AND (available-reserved) >= 1;

if (rows_affected == 0) {
  # Try next bucket
  for i in range(1, 10):
    next_bucket = (bucket_id + i) % 10;
    retry UPDATE on next_bucket;
    if success: break;
}
Interview Tip: Always explain the atomic check-and-set pattern. Saying "just check stock > 0 then decrement" has a race condition between check and decrement. The WHERE clause with the stock check MUST be inside the UPDATE statement to be atomic. For flash sales, mention bucketing to avoid single-row lock bottleneck. Interviewers love this detail because it shows you understand high-concurrency systems.
Key Point: The reservation system is the secret sauce. It prevents overselling (atomic check-and-set) while allowing users time to complete payment (15min TTL). Without reservations, you'd either oversell (check then update) or block inventory indefinitely (no expiration). The background job that releases expired reservations is critical for reclaiming abandoned cart inventory.