Chapter 54: Design Stripe/Payment System

1. Requirements

Functional Requirements

Requirement Description Priority
Process Payments Accept credit/debit cards, authorize, capture, settle transactions P0
Idempotency Prevent duplicate charges from retries using idempotency keys P0
Multi-Currency Support 135+ currencies with real-time FX conversion P0
Refunds & Disputes Full/partial refunds, chargeback handling, dispute management P0
Webhooks Event notifications to merchants (payment succeeded/failed/refunded) P0
Payment Methods Cards, ACH, wire transfer, digital wallets (Apple Pay, Google Pay) P1
Subscriptions Recurring billing with retry logic for failed payments P1
Analytics Dashboard Transaction history, revenue reports, settlement tracking P2

Non-Functional Requirements

Requirement Target Interview Callout
Availability 99.999% (5 min/year downtime) Multi-region active-active, no single point of failure
Consistency Strong consistency for ledger, eventual for analytics Money is never lost or double-charged
Latency p99 < 500ms for API, < 3s for payment auth User experience critical at checkout
PCI Compliance PCI DSS Level 1 certification Tokenization to reduce scope, no raw card storage
Durability Zero transaction data loss Append-only ledger, replicated across regions
Security Encryption at rest/transit, audit logs, fraud detection TLS 1.3, AES-256, HMAC signature verification

Out of Scope

Fraud detection ML models (assume provided by third-party service)
Merchant onboarding/KYC (Know Your Customer) workflows
Tax calculation and compliance (integrate with TaxJar/Avalara)

2. Capacity Estimation

Assumptions (Stripe Scale)

Metric Value Calculation
Daily Active Merchants 2 million Stripe processes for millions of businesses
Transactions per Day 500 million 2M merchants × 250 txns/day average
Transactions per Second (TPS) ~6,000 avg, ~18,000 peak 500M / 86,400 sec × 3x peak multiplier
Read:Write Ratio 10:1 Dashboard queries, webhook retries, reconciliation checks
Avg Transaction Size 2 KB Payment intent metadata, card token, merchant ID, amounts
Daily Storage 1 TB/day 500M × 2 KB = 1 TB (append-only ledger)
5-Year Storage 1.8 PB 1 TB × 365 × 5 = 1.8 PB (with compression)
Webhook Events 1.5 billion/day 500M txns × 3 events avg (created, succeeded, settled)
Reconciliation Jobs Hourly Match 20M txns/hour against bank statements

Network Bandwidth

Direction Calculation Result
Ingress (Writes) 6,000 TPS × 2 KB 12 MB/s (~100 Mbps)
Egress (Reads) 60,000 QPS × 2 KB 120 MB/s (~1 Gbps)
Webhooks 17,000 events/sec × 1 KB 17 MB/s (~140 Mbps)
Interview Tip: Always clarify scale early. A payment system for a small startup (10 TPS) has dramatically different requirements than Stripe (18,000 peak TPS). Ask: "What transaction volume are we targeting?"

3. High-Level Design

graph TB Client[Client/Merchant App] subgraph API Gateway Layer LB[Load Balancer
TLS Termination] AG[API Gateway
Rate Limiting, Auth] end subgraph Core Services PS[Payment Service
State Machine] IS[Idempotency Service
Key Deduplication] LS[Ledger Service
Double-Entry Bookkeeping] WS[Webhook Service
Event Delivery] RS[Reconciliation Service
Bank Matching] TS[Tokenization Service
PCI Vault] end subgraph Data Stores PSQL[(PostgreSQL
Payment Intents)] Ledger[(Ledger DB
Immutable Log)] Redis[(Redis
Idempotency Cache)] Queue[Kafka/SQS
Event Queue] end subgraph External Systems PSP1[PSP: Visa/Mastercard] PSP2[PSP: Bank Network] PSP3[Backup PSP] Bank[Bank Statements API] end Client -->|HTTPS| LB LB --> AG AG --> PS AG --> IS PS --> PSQL PS --> LS PS --> TS PS -->|Route to PSP| PSP1 PS -->|Fallback| PSP2 PS -->|Fallback| PSP3 IS --> Redis LS --> Ledger PS -->|Publish Events| Queue Queue --> WS Queue --> RS WS -->|HTTP POST| Client RS --> Bank RS --> Ledger TS -->|Store Tokens| PSQL

Component Responsibilities

Component Responsibility Technology Choice
API Gateway Authentication (API keys), rate limiting, request validation Kong, AWS API Gateway, custom Go service
Payment Service Orchestrate payment flow, manage state machine, route to PSPs Java/Go microservice, horizontally scaled
Idempotency Service Check idempotency keys, return cached responses for duplicates Redis with 24-hour TTL, fallback to PostgreSQL
Ledger Service Record every financial transaction as double-entry, immutable append PostgreSQL with append-only constraint, sharded by merchant_id
Webhook Service Deliver events to merchant endpoints with retry logic Distributed queue (Kafka), worker pool, exponential backoff
Reconciliation Service Match internal ledger with bank statements, detect discrepancies Batch job (Airflow), runs hourly, flags mismatches for review
Tokenization Service Convert raw card numbers to tokens, store in PCI-compliant vault HSM (Hardware Security Module), AES-256 encryption
PSP Integration Communicate with card networks (Visa/MC), handle timeouts/retries ISO 8583 protocol for card networks, HTTP for modern PSPs
Interview Callout: Explain why you chose specific databases. PostgreSQL for ledger (ACID, strong consistency), Redis for idempotency (fast lookups, TTL), Kafka for events (high throughput, replay capability).

4. Payment Flow (Two-Phase Commit)

Authorize → Capture → Settle

Phase Action State Transition Timing
1. Authorize Hold funds on customer's card (pending charge) CREATED → AUTHORIZED Immediate (~2s)
2. Capture Confirm the charge, move from "hold" to actual debit AUTHORIZED → CAPTURED Within 7 days (merchant triggers)
3. Settle Bank transfers funds from customer to merchant account CAPTURED → SETTLED T+2 days (batch settlement)

Why Two-Phase?

Inventory Reservation: Hotel books room during authorize, captures when guest checks in
Fraud Prevention: Merchant reviews high-value orders before capturing
Partial Captures: Authorize $100, capture $80 if item out of stock
Authorization Expiry: Holds expire after 7 days, requiring reauthorization

Sequence Diagram

sequenceDiagram participant Client participant API participant Payment Service participant Idempotency participant Ledger participant PSP participant Webhook Client->>API: POST /v1/payment_intents
{amount, currency, card_token}
Idempotency-Key: abc123 API->>Idempotency: Check key "abc123" Idempotency-->>API: Not found (new request) API->>Payment Service: Create PaymentIntent Payment Service->>Ledger: Record CREATED state Payment Service->>PSP: Authorize $50.00 PSP-->>Payment Service: Authorization approved (auth_code: XYZ) Payment Service->>Ledger: Record AUTHORIZED state + auth_code Payment Service->>Idempotency: Cache response for "abc123" (24h TTL) Payment Service-->>API: {id: pi_123, status: authorized} API-->>Client: 200 OK Payment Service->>Webhook: Publish event: payment_intent.authorized Webhook->>Client: POST merchant.com/webhook
{event: payment_intent.authorized} Note over Client,Webhook: --- Later: Merchant captures payment --- Client->>API: POST /v1/payment_intents/pi_123/capture API->>Payment Service: Capture pi_123 Payment Service->>PSP: Capture authorization XYZ PSP-->>Payment Service: Capture confirmed Payment Service->>Ledger: Record CAPTURED state Payment Service->>Ledger: Double-entry: Debit Customer, Credit Merchant Payment Service->>Webhook: Publish event: payment_intent.captured Webhook->>Client: POST merchant.com/webhook

State Machine

From State Event To State Ledger Action
- Create payment intent CREATED Log intent creation
CREATED PSP authorizes AUTHORIZED Log authorization hold
AUTHORIZED Merchant captures CAPTURED Debit customer, credit merchant
CAPTURED Bank settles SETTLED Mark as reconciled
AUTHORIZED Cancel before capture CANCELED Release hold, log cancellation
CAPTURED Refund issued REFUNDED Credit customer, debit merchant
CREATED PSP declines FAILED Log failure reason
Interview Callout: Emphasize that state transitions are atomic and logged to the ledger. If a PSP call times out, the payment stays in the previous state until reconciliation determines the actual outcome.

5. Idempotency

Problem: Network Retries Cause Double Charges

Scenario: Client sends payment request, network times out, client retries
Without idempotency: Server processes both requests → customer charged twice
With idempotency: Server detects duplicate key, returns cached response

Idempotency Key Design

Aspect Implementation Rationale
Key Format UUID v4 or merchant-generated unique string Client controls uniqueness, server validates format
Scope Per merchant account Key "abc123" for Merchant A doesn't collide with Merchant B
Storage Redis (primary), PostgreSQL (fallback) Redis fast lookup, DB for durability if Redis evicts
TTL 24 hours Balance between preventing duplicates and cache size
Response Caching Store full HTTP response body + status code Return identical response to client on retry
In-Progress Handling If request still processing, return 409 Conflict Client should retry with same key after delay

Flow with Idempotency

graph LR A[Client Request
Idempotency-Key: X] --> B{Key in Redis?} B -->|Yes, Complete| C[Return Cached Response] B -->|Yes, In-Progress| D[Return 409 Conflict] B -->|No| E[Mark Key as In-Progress] E --> F[Process Payment] F --> G[Store Response in Redis] G --> H[Return Response to Client]

Edge Cases

Edge Case Handling
Same key, different payload Reject with 422 Unprocessable Entity (key reuse forbidden)
Server crashes mid-processing Key stuck as "in-progress", timeout after 60s, retry allowed
Redis evicts key early Fallback to DB lookup, if not found treat as new request
Client never retries Key expires after 24h, no issue (one-time payment)
Interview Callout: Idempotency is critical for payment systems. Always ask: "What happens if the client retries?" Store not just the payment ID, but the full response to ensure identical behavior.

6. Ledger & Double-Entry Bookkeeping

Double-Entry Accounting Principle

Every transaction has two sides: Debit (money leaving) and Credit (money arriving)
Sum of debits = Sum of credits: System is always balanced, catch bugs early
Immutable log: Entries never deleted/updated, corrections via new entries

Example: $100 Payment from Customer to Merchant

Account Debit Credit Description
Customer Card Account $100 - Money leaving customer
Merchant Revenue Account - $97 Merchant receives $97 (after fees)
Stripe Fee Account - $3 Stripe earns $3 fee (3%)
Total $100 $100 Balanced

Ledger Schema

Column Type Description
entry_id UUID Unique ID for each ledger entry
transaction_id UUID Groups debit + credit entries (payment_intent_id)
account_id String Customer/merchant/Stripe account identifier
debit_amount Decimal(19,4) Amount debited (or NULL if credit entry)
credit_amount Decimal(19,4) Amount credited (or NULL if debit entry)
currency String(3) ISO 4217 code (USD, EUR, GBP)
timestamp Timestamp When entry was created (immutable)
description Text Human-readable description (e.g., "Payment from John")
idempotency_key String Link back to original API request

Ledger Architecture Diagram

graph TB subgraph Payment Flow PS[Payment Service] end subgraph Ledger Service LS[Ledger Service
Validates Double-Entry] Check{Sum Debits =
Sum Credits?} end subgraph Storage PG[(PostgreSQL
Append-Only Table)] Archive[(Cold Storage
S3 for old entries)] end subgraph Reconciliation Recon[Reconciliation Job
Hourly] Bank[Bank Statement API] end PS -->|Submit Transaction| LS LS --> Check Check -->|Yes| PG Check -->|No| Error[Reject Transaction
Log Alert] PG -->|Archive entries > 1 year| Archive Recon --> PG Recon --> Bank Recon -->|Flag Discrepancies| Alert[Alert System]

Refund Handling (Correction Entry)

Original Payment Debit Credit
Customer Card $100 -
Merchant Revenue - $100
Refund Transaction Debit Credit
Merchant Revenue $100 -
Customer Card - $100
Interview Callout: Never UPDATE or DELETE ledger entries. Refunds/corrections are new entries that reverse the original. This creates an audit trail and prevents data loss.

7. PSP (Payment Service Provider) Integration

What is a PSP?

Definition: Intermediary between merchants and card networks (Visa, Mastercard)
Examples: Stripe connects to multiple PSPs (Worldpay, Adyen, Chase Paymentech)
Role: Route payment to issuing bank, handle auth/capture/settle, return success/failure

PSP Integration Challenges

Challenge Solution Interview Callout
Timeout Handling Set 30s timeout, if no response assume unknown state Query PSP's status API to confirm final state
PSP Downtime Route to backup PSP (circuit breaker pattern) Monitor PSP health, failover if success rate < 95%
Retry Logic Retry transient errors (5xx), not permanent (card declined) Exponential backoff: 1s, 2s, 4s, 8s (max 3 retries)
Rate Limits PSP may limit to 1000 TPS per merchant Queue requests, apply token bucket rate limiting
Multi-Currency Different PSPs support different regions/currencies Route EUR to European PSP, USD to US PSP
Protocol Variance Old PSPs use ISO 8583, modern use JSON REST APIs Adapter pattern to normalize PSP responses

Circuit Breaker Pattern

stateDiagram-v2 [*] --> Closed Closed --> Open: Failure rate > 50%
(5+ failures in 10s) Open --> HalfOpen: After 60s cooldown HalfOpen --> Closed: Test request succeeds HalfOpen --> Open: Test request fails note right of Closed: Normal operation, All requests sent to PSP note right of Open: Stop sending requests, Return cached error, Route to backup PSP note right of HalfOpen: Send 1 test request, If success resume normal

PSP Routing Strategy

Criteria Primary PSP Backup PSP Reason
Currency: USD Chase Paymentech Worldpay Lowest fees for US transactions
Currency: EUR Adyen Checkout.com European PSPs have better rates for EUR
High-Risk Merchant Specialized PSP - Gambling/pharma require specific PSPs
3D Secure Required PSP with 3DS support - EU Strong Customer Authentication (SCA)

Timeout Recovery Flow

Step Action Reasoning
1 Send authorize request to PSP, 30s timeout Standard payment authorization
2 Timeout occurs, no response from PSP Network issue or PSP overloaded
3 Mark payment as PENDING, return 202 Accepted to client Honest response: "We don't know yet"
4 Background job queries PSP status API every 10s Poll for actual result (authorized/declined)
5 PSP returns final status → update payment state Reconcile actual outcome
6 Send webhook to merchant with final result Notify merchant of resolution
Interview Callout: Never assume a timeout means failure. The PSP may have processed the payment but the response was lost. Always query the PSP's status endpoint to confirm.

8. Reconciliation

What is Reconciliation?

Definition: Match internal ledger against external bank statements to ensure consistency
Frequency: Hourly for recent transactions, daily for settled transactions
Goal: Detect discrepancies (missing payments, double charges, incorrect amounts)

Reconciliation Flow

graph TB subgraph Internal Systems Ledger[(Ledger DB
Expected Transactions)] end subgraph External Systems Bank[Bank Statements API
Actual Settlements] end subgraph Reconciliation Service Fetch[Fetch Transactions
Last Hour] Match{Match by
transaction_id
+ amount?} Resolve[Resolve Discrepancies] end subgraph Outcomes Success[Mark as RECONCILED] Missing[Missing in Bank
→ Query PSP Status] Extra[Extra in Bank
→ Flag for Review] Mismatch[Amount Mismatch
→ Alert Finance Team] end Ledger --> Fetch Bank --> Fetch Fetch --> Match Match -->|Found in Both| Success Match -->|In Ledger, Not Bank| Missing Match -->|In Bank, Not Ledger| Extra Match -->|Amounts Differ| Mismatch Missing --> Resolve Extra --> Resolve Mismatch --> Resolve

Reconciliation Logic

Discrepancy Type Root Cause Resolution
In Ledger, Not in Bank Payment authorized but not captured, or capture failed Query PSP status, if failed → mark as FAILED, refund customer
In Bank, Not in Ledger PSP sent settlement, but webhook never arrived Create ledger entry retroactively, investigate webhook failure
Amount Mismatch Currency conversion issue, or fee calculation error Create adjustment entry in ledger, alert finance team
Duplicate in Bank PSP processed payment twice (PSP bug) Initiate chargeback to PSP, refund customer
Delayed Settlement Bank statement lags by T+2 days Normal, retry reconciliation after 3 days

Reconciliation Job Schedule

Job Type Frequency Scope Purpose
Real-Time Per transaction Single payment Immediate feedback on PSP response
Hourly Every hour Last 1 hour Catch timeouts, delayed PSP responses
Daily 3 AM UTC Previous day Match against bank statements (T+2)
Weekly Sunday 1 AM Previous 7 days Final cleanup, detect long-delayed issues

Alerting Thresholds

Metric Threshold Action
Discrepancy Rate > 0.1% Page on-call engineer (indicates systemic issue)
Missing Transactions > 10 in an hour Investigate PSP connection, possible downtime
Amount Mismatches > $1000 cumulative Alert finance team, potential fee calculation bug
Reconciliation Job Failure Any failure Page immediately (blind to discrepancies)
Interview Callout: Reconciliation is not optional—it's how you detect silent failures. A payment system without reconciliation will slowly leak money through undetected errors.

9. Webhook System

Why Webhooks?

Push vs Pull: Merchants get instant notifications instead of polling for status
Events: payment_intent.succeeded, payment_intent.failed, refund.created, etc.
Delivery Guarantee: At-least-once (may deliver duplicates, merchants must deduplicate)

Webhook Architecture

graph LR PS[Payment Service] -->|Publish Event| Queue[Kafka/SQS Queue] Queue --> Worker1[Webhook Worker 1] Queue --> Worker2[Webhook Worker 2] Queue --> Worker3[Webhook Worker N] Worker1 -->|POST /webhook| Merchant1[Merchant A Endpoint] Worker2 -->|POST /webhook| Merchant2[Merchant B Endpoint] Worker3 -->|POST /webhook| Merchant3[Merchant C Endpoint] Merchant1 -->|200 OK| Worker1 Merchant1 -->|5xx Error| Retry[Retry Queue
Exponential Backoff] Retry -->|Retry 1: 10s| Worker1 Retry -->|Retry 2: 30s| Worker1 Retry -->|Retry 3: 1m| Worker1 Retry -->|Retry 10: 24h| DLQ[Dead Letter Queue
Manual Review]

Webhook Delivery Logic

Step Action Details
1. Publish Event Payment service publishes to Kafka Event payload includes: event_id, type, timestamp, data
2. Worker Consumes Webhook worker reads from queue Scalable worker pool (autoscale based on queue depth)
3. Sign Payload Generate HMAC-SHA256 signature Signature = HMAC(secret, event_payload), sent in header
4. Send HTTP POST POST to merchant endpoint with 30s timeout Include signature in Stripe-Signature header
5. Check Response 2xx = success, 5xx = retry, 4xx = discard Merchant should return 200 within 30s
6. Retry Logic Exponential backoff: 10s, 30s, 1m, 5m, ... 24h Max 10 retries over 3 days
7. Dead Letter Queue After 10 failures, move to DLQ Operations team manually investigates, contacts merchant

Signature Verification (Merchant Side)

Step Implementation
1. Extract Header Read Stripe-Signature header from HTTP request
2. Parse Signature Format: t=1234567890,v1=abc123..., extract timestamp + signature
3. Compute Expected expected_sig = HMAC-SHA256(webhook_secret, timestamp + "." + body)
4. Compare If expected_sig == received_sig, webhook is authentic
5. Check Timestamp Reject if timestamp > 5 minutes old (replay attack prevention)

Webhook Event Types

Event Type Trigger Merchant Action
payment_intent.created Payment intent created Log for tracking
payment_intent.succeeded Payment captured successfully Fulfill order, ship product
payment_intent.payment_failed Card declined or PSP error Notify customer, prompt for new payment method
charge.refunded Refund issued Cancel shipment, update order status
charge.dispute.created Customer initiated chargeback Provide evidence to Stripe within 7 days
payout.paid Funds transferred to merchant bank account Update cash flow forecast

Idempotency for Webhooks (Merchant Responsibility)

Problem: Webhook may be delivered multiple times (network retry)
Solution: Merchant stores event_id, skips processing if already seen
Example: If "payment_succeeded" webhook arrives twice, only fulfill order once
Interview Callout: Webhooks are at-least-once, not exactly-once. Merchants must handle duplicates. Always include event_id in payload so merchants can deduplicate.

10. Security & PCI Compliance

PCI DSS (Payment Card Industry Data Security Standard)

Requirement Implementation Stripe's Approach
Never Store Raw Card Numbers Tokenization: Replace card with token immediately Stripe.js collects card, sends directly to Stripe (bypasses merchant)
Encryption at Rest AES-256 encryption for all sensitive data Tokens encrypted in HSM (Hardware Security Module)
Encryption in Transit TLS 1.3 for all API communication Enforce HTTPS, reject HTTP, certificate pinning
Network Segmentation Isolate cardholder data environment (CDE) Tokenization vault in separate VPC, no internet access
Access Control Least privilege, MFA for all employees Role-based access, audit logs for every action
Regular Audits Annual PCI audit by Qualified Security Assessor (QSA) Stripe maintains Level 1 PCI DSS compliance

Tokenization Flow

sequenceDiagram participant Browser participant StripeJS as Stripe.js Client-Side participant TokenVault as Tokenization Service - PCI Scope participant MerchantAPI as Merchant Backend - Non-PCI Browser->>StripeJS: User enters card: 4242 4242 4242 4242 StripeJS->>TokenVault: POST /v1/tokens
{card: 4242...} Note over TokenVault: Encrypt card in HSM
Generate token TokenVault-->>StripeJS: {token: tok_abc123} StripeJS->>MerchantAPI: POST /checkout
{token: tok_abc123} Note over MerchantAPI: Merchant never sees
raw card number MerchantAPI->>TokenVault: POST /v1/charges
{token: tok_abc123, amount: 1000} TokenVault-->>MerchantAPI: {charge: ch_xyz, status: succeeded}

Security Layers

Layer Mechanism Threat Mitigated
API Authentication Secret API keys (sk_live_...), rotated quarterly Unauthorized access to merchant accounts
Rate Limiting 100 requests/sec per merchant, 1000/sec global DDoS attacks, brute force
Fraud Detection Radar ML model scores each transaction (0-100 risk) Stolen cards, synthetic identity fraud
3D Secure (3DS) Redirect to bank for additional auth (password/SMS) Card-not-present fraud (EU SCA requirement)
CVV Verification Validate 3-digit CVV code, never stored Card-present → card-not-present attacks
Address Verification (AVS) Match billing address with card issuer records Stolen card with incorrect address
Webhook Signatures HMAC-SHA256 prevents payload tampering Man-in-the-middle, replay attacks
Audit Logging Immutable logs of every API call, admin action Insider threats, forensic investigations

PCI Scope Reduction

Without Tokenization With Tokenization
Merchant stores card numbers Merchant only stores tokens
Entire merchant infrastructure in PCI scope Only Stripe's tokenization vault in PCI scope
Annual PCI audit costs $50k-$500k Merchant fills out simple SAQ-A form (self-assessment)
Breach of merchant DB exposes millions of cards Breach exposes useless tokens (can't be reused)
Interview Callout: Tokenization is the key to PCI compliance. By never touching raw card numbers, merchants dramatically reduce their compliance burden and risk.

11. Scalability & Availability

Horizontal Scaling Strategy

Component Scaling Approach Bottleneck Mitigation
API Gateway Stateless, auto-scale based on CPU (target: 70%) Add instances in 30 seconds during traffic spikes
Payment Service Stateless, scale to 1000+ instances Partition by merchant_id hash for cache locality
Ledger Service Stateless writers, append-only log Shard PostgreSQL by merchant_id (100 shards)
Webhook Workers Auto-scale based on Kafka queue depth Target: queue lag < 10 seconds
PostgreSQL Vertical scaling (large instances) + read replicas Route reads to replicas, writes to primary
Redis Redis Cluster with 100+ nodes Hash slot sharding, replicas for failover
Kafka Partitioned by merchant_id (1000 partitions) Parallel webhook delivery per partition

Multi-Region Architecture (Active-Active)

graph TB subgraph US Region LB1[Load Balancer] API1[API Gateway] PS1[Payment Service] PG1[(PostgreSQL Primary)] Redis1[(Redis Cluster)] end subgraph EU Region LB2[Load Balancer] API2[API Gateway] PS2[Payment Service] PG2[(PostgreSQL Primary)] Redis2[(Redis Cluster)] end subgraph APAC Region LB3[Load Balancer] API3[API Gateway] PS3[Payment Service] PG3[(PostgreSQL Primary)] Redis3[(Redis Cluster)] end Client1[US Client] -->|Low Latency| LB1 Client2[EU Client] -->|Low Latency| LB2 Client3[APAC Client] -->|Low Latency| LB3 PG1 -.->|Async Replication| PG2 PG2 -.->|Async Replication| PG3 PG3 -.->|Async Replication| PG1 Redis1 -.->|Idempotency Sync| Redis2 Redis2 -.->|Idempotency Sync| Redis3

Database Sharding Strategy

Aspect Design Decision Rationale
Shard Key merchant_id Most queries scoped to single merchant, no cross-shard joins
Shard Count 100 shards (expandable to 1000) Balance between overhead and future growth
Routing Consistent hashing: shard = hash(merchant_id) % 100 Even distribution, minimal rebalancing when adding shards
Replication 3 replicas per shard (1 primary, 2 read replicas) Survive 2 failures, read scaling
Cross-Shard Queries Analytics DB (Redshift) replicates from all shards OLTP shards optimized for writes, OLAP for aggregations

Event Sourcing for Audit Trail

Definition: Store every state change as an event, reconstruct current state by replaying events
Benefit: Complete history, can replay to debug issues, audit compliance
Example: payment_intent_created → payment_authorized → payment_captured → payment_settled
Storage: Kafka topic with infinite retention (compaction for snapshots)

Availability Tactics

Tactic Implementation Impact on Availability
No Single Point of Failure Every component replicated 3x across AZs Survive zone failure (3% → 0.001% failure rate)
Health Checks Load balancer pings /health every 5s, removes unhealthy instances Automatic failover in 15 seconds
Circuit Breakers Stop calling failed PSPs, route to backup Prevent cascading failures
Graceful Degradation If fraud check times out, allow payment (review later) Maintain core functionality during partial outage
Database Failover Automatic promotion of replica to primary in 30s Recover from primary failure quickly
Chaos Engineering Randomly kill 1% of instances daily (Chaos Monkey) Validate resilience continuously

Capacity Planning

Metric Current Peak Headroom
API Instances 100 300 (Black Friday) Auto-scale to 500 max
Database Connections 1000/shard 3000/shard PgBouncer connection pooling
Kafka Throughput 50k events/sec 150k events/sec Provision for 200k/sec
Webhook Workers 500 2000 Auto-scale based on queue lag
Interview Callout: 99.999% availability means only 5 minutes of downtime per year. Achieve this through multi-region active-active deployment, eliminating all single points of failure, and aggressive failover automation.

Summary: Key Interview Talking Points

Topic Key Point
Idempotency Use idempotency keys to prevent double charges on retries, cache responses for 24h
Two-Phase Commit Authorize holds funds, capture charges them (within 7 days), settle transfers to merchant (T+2)
Ledger Double-entry bookkeeping ensures sum(debits) = sum(credits), immutable append-only log
PSP Integration Circuit breaker pattern for failover, query status API on timeout (never assume failure)
Reconciliation Match internal ledger with bank statements hourly, detect and resolve discrepancies
Webhooks At-least-once delivery, exponential backoff retry, HMAC signatures for security
PCI Compliance Tokenization removes merchant from PCI scope, never store raw card numbers
Scalability Shard by merchant_id, multi-region active-active, event sourcing for audit trail
Availability 99.999% via no single points of failure, automatic failover, chaos engineering

Clarifying Questions to Ask in Interview

1. What transaction volume are we targeting? (10 TPS vs 10,000 TPS changes design)
2. Are we building the full PSP or integrating with existing PSPs?
3. What payment methods? (Cards only vs ACH, wire, crypto)
4. Multi-currency support? (FX conversion, regional PSPs)
5. Is this B2C or B2B? (B2B may need invoicing, net-30 terms)
6. What's the acceptable latency? (Checkout should be <3s end-to-end)
7. Are we responsible for fraud detection or integrating third-party?

Operational Readiness (Production-Grade)

Service Level Indicators (SLIs)

Monitoring & Alerting

Deployment Strategy

Circuit Breakers

Disaster Recovery

Testing