Chapter 73: Multi-Tenancy Architecture

Multi-tenancy is the architectural approach where a single system serves multiple customers (tenants), each isolated from the others. It's fundamental to SaaS platforms like Salesforce, Slack, and Shopify. The core challenge: balance cost efficiency (shared resources) with isolation (security, performance, customization). This chapter covers isolation models, resource management, data security, and scaling strategies.

Isolation Models

There are four main isolation models, each with different trade-offs between cost, security, and operational complexity.

1. Shared Everything (Pool Model)

graph TB
    subgraph "Single Application Instance"
        App[Application Server]
    end

    subgraph "Single Database"
        DB[(Database)]
        T1[Tenant A data
tenant_id = 'A']
        T2[Tenant B data
tenant_id = 'B']
        T3[Tenant C data
tenant_id = 'C']
    end

    App --> DB
    DB --> T1
    DB --> T2
    DB --> T3

    style T1 fill:#ffcccc
    style T2 fill:#ccffcc
    style T3 fill:#ccccff

Characteristics

  • All tenants share same app servers and database
  • Data distinguished by tenant_id column
  • Every query must include: WHERE tenant_id = 'X'

Pros

  • Lowest cost: Maximum resource sharing
  • Operational simplicity: One codebase, one database to manage
  • Easiest onboarding: New tenant is just a row

Cons

  • Security risk: One bug exposes all tenant data
  • Noisy neighbor: One tenant's heavy load affects all
  • Limited customization: All tenants get same schema
  • Compliance challenges: Hard to isolate for regulations (GDPR, HIPAA)

Best For

  • Startups optimizing for cost
  • B2C SaaS with homogeneous tenants
  • Low security/compliance requirements

2. Shared Database, Separate Schema

graph TB
    subgraph "Application Layer"
        App[Application Server
Multi-schema aware]
    end

    subgraph "Single Database Instance"
        DB[(PostgreSQL)]
        SchemaA[Schema: tenant_a
users, orders, products]
        SchemaB[Schema: tenant_b
users, orders, products]
        SchemaC[Schema: tenant_c
users, orders, products]
    end

    App --> |Connection pool A| SchemaA
    App --> |Connection pool B| SchemaB
    App --> |Connection pool C| SchemaC

    DB --> SchemaA
    DB --> SchemaB
    DB --> SchemaC

    style SchemaA fill:#ffcccc
    style SchemaB fill:#ccffcc
    style SchemaC fill:#ccccff

Characteristics

  • Each tenant has its own database schema
  • Queries: SELECT * FROM tenant_a.users
  • Supports per-tenant schema customization

Pros

  • Better isolation: Accidental cross-tenant query fails
  • Customization: Each tenant can have custom columns
  • Easier migration: Export one schema = one tenant
  • Resource limits: Per-schema quotas (PostgreSQL)

Cons

  • Schema sprawl: Thousands of schemas in one DB
  • Migration complexity: Schema changes × N tenants
  • Still noisy neighbor: Shared CPU, memory, disk
  • Connection limits: Each schema needs connection pool

Best For

  • B2B SaaS with moderate tenant count (10s-100s)
  • Need for per-tenant customization
  • Medium security requirements

3. Separate Database per Tenant

graph TB
    subgraph "Application Layer"
        App[Application Server
Database router]
    end

    subgraph "Shared Database Server"
        Server[PostgreSQL Server]
        DBA[(Database: tenant_a)]
        DBB[(Database: tenant_b)]
        DBC[(Database: tenant_c)]
    end

    App --> |Route based on tenant_id| Server
    Server --> DBA
    Server --> DBB
    Server --> DBC

    style DBA fill:#ffcccc
    style DBB fill:#ccffcc
    style DBC fill:#ccccff

Characteristics

  • Each tenant has its own database
  • Application routes requests to correct database
  • Strong logical isolation

Pros

  • Strong isolation: Query cannot access other tenant's data
  • Independent backups: Restore one tenant without affecting others
  • Performance isolation: Slow query affects only that tenant
  • Easier compliance: Can encrypt/isolate specific tenants

Cons

  • Database sprawl: Hundreds of databases to manage
  • Higher cost: Each DB has overhead (connections, buffers)
  • Complex migrations: Deploy schema change × N databases
  • Still noisy neighbor: Shared server CPU, memory, disk

Best For

  • Regulated industries (healthcare, finance)
  • High-value enterprise customers
  • Need strong isolation without full infrastructure separation

4. Separate Infrastructure (Silo Model)

graph TB
    subgraph "Tenant A Infrastructure"
        AppA[App Servers]
        DBA[(Database)]
        CacheA[Redis]
    end

    subgraph "Tenant B Infrastructure"
        AppB[App Servers]
        DBB[(Database)]
        CacheB[Redis]
    end

    subgraph "Tenant C Infrastructure"
        AppC[App Servers]
        DBC[(Database)]
        CacheC[Redis]
    end

    LB[Load Balancer
Tenant Router]

    User[Users] --> LB
    LB --> AppA
    LB --> AppB
    LB --> AppC

    style AppA fill:#ffcccc
    style AppB fill:#ccffcc
    style AppC fill:#ccccff

Characteristics

  • Each tenant gets dedicated app servers, database, cache
  • Completely isolated infrastructure
  • Can be on-prem or separate AWS accounts/VPCs

Pros

  • Maximum isolation: No resource contention
  • Custom versions: Each tenant can run different code version
  • Compliance friendly: Easy to meet data residency requirements
  • Performance guarantees: No noisy neighbor

Cons

  • Highest cost: No resource sharing
  • Operational nightmare: Manage N complete stacks
  • Slow onboarding: Provisioning new tenant takes hours
  • Difficult updates: Deploy to hundreds of environments

Best For

  • Enterprise customers paying premium
  • Strict compliance (government, healthcare)
  • Need custom code per tenant

Isolation Model Comparison

Aspect Shared Everything Separate Schema Separate Database Separate Infra
Cost Lowest Low-Medium Medium-High Highest
Security Weak Medium Strong Maximum
Noisy Neighbor High risk High risk Medium risk No risk
Customization Limited Per-schema Per-database Fully custom
Onboarding Instant Seconds Minutes Hours
Tenant Limit Millions Thousands Hundreds Tens
Operations Simple Moderate Complex Very complex
Interview Tip: Explain the spectrum from shared (cost-efficient) to isolated (secure). Most companies use a hybrid: shared everything for small tenants, separate infra for enterprise. Mention the "tier-based" approach: free tier = shared, premium = separate DB, enterprise = separate infra. Always discuss the noisy neighbor problem and how isolation level mitigates it.

Noisy Neighbor Problem

The noisy neighbor problem occurs when one tenant's resource consumption impacts other tenants' performance. It's the primary downside of resource sharing in multi-tenant systems.

Manifestations

1. CPU Contention

  • Scenario: Tenant A runs expensive analytics query
  • Impact: Tenant B's simple queries become slow
  • Example: API latency spikes from 50ms → 5s

2. Memory Pressure

  • Scenario: Tenant A caches 10GB of data
  • Impact: Database starts swapping, all tenants slow down
  • Example: Query latency increases due to disk I/O

3. Disk I/O Saturation

  • Scenario: Tenant A imports 1TB of data
  • Impact: Disk IOPS consumed, all queries wait
  • Example: Dashboard takes 30s to load instead of 2s

4. Connection Exhaustion

  • Scenario: Tenant A opens 500 database connections
  • Impact: Connection pool exhausted, other tenants get errors
  • Example: "Too many connections" error

5. Queue Backlog

  • Scenario: Tenant A enqueues 1M background jobs
  • Impact: Other tenants' jobs delayed for hours
  • Example: Email not sent for 3 hours

Detection

graph TB
    subgraph "Monitoring System"
        M1[Per-Tenant Metrics
CPU, Memory, IOPS, Query time]
        M2[Anomaly Detection
Spike detection]
        M3[Correlation Analysis
When Tenant A heavy, Tenant B slow]
    end

    subgraph "Alerting"
        A1[Alert: Tenant A using 80% CPU]
        A2[Alert: p99 latency spike for Tenant B]
        A3[Correlation: Tenant A heavy load = Tenant B degradation]
    end

    M1 --> M2
    M2 --> A1
    M2 --> A2
    M1 --> M3
    M3 --> A3

Mitigation Strategies

1. Resource Quotas (see next section)

Limit per-tenant CPU, memory, IOPS, connections.

2. Rate Limiting

  • API rate limits: 1000 requests/min per tenant
  • Query rate limits: 100 queries/sec per tenant
  • Background job rate: 10 jobs/sec per tenant

3. Query Timeout

  • Kill queries running longer than 30 seconds
  • Prevents one tenant's runaway query blocking others

4. Priority Queues

  • Enterprise tenants get high-priority queue
  • Free tenants in low-priority queue
  • Ensures premium customers unaffected

5. Tenant Sharding

  • Separate heavy tenants onto dedicated shards
  • Small tenants colocated, large tenants isolated

6. Circuit Breakers

  • Temporarily block tenant exceeding quota
  • Return "503 Service Unavailable" until cooldown

Real-World Example: AWS Lambda

Problem: Lambda is multi-tenant. How does AWS prevent noisy neighbors?

AWS Solutions

  • Firecracker microVMs: Strong isolation between functions
  • Concurrency limits: Max 1000 concurrent executions per account (default)
  • Reserved concurrency: Guarantee capacity for critical functions
  • Memory limits: Each function has max memory (128MB-10GB)
  • Execution time limit: Max 15 minutes, prevents runaway functions
  • Regional quotas: Limits per region, not global
Interview Tip: The noisy neighbor problem is a favorite interview topic for distributed systems. Explain with a concrete example (e.g., "Tenant A runs a report, database CPU spikes, Tenant B's dashboard times out"). Discuss both detection (per-tenant metrics, anomaly detection) and mitigation (quotas, rate limiting, sharding). Mention that you can't fully eliminate it in shared systems - only reduce impact.

Resource Quotas

Resource quotas limit per-tenant consumption of CPU, memory, disk, IOPS, connections, and other resources. They're essential for fair resource allocation and preventing noisy neighbors.

Types of Quotas

Resource Quota Type Implementation Example
API Requests Rate limit Token bucket, sliding window 1000 req/min
Storage Hard limit Filesystem quotas, DB monitoring 100 GB per tenant
Database IOPS Soft limit Throttle queries when exceeded 1000 IOPS
Compute (CPU) Fair share Cgroups, Kubernetes limits 20% of CPU
Memory Hard limit Cgroups, container limits 4 GB RAM
Connections Pool limit Connection pool per tenant 50 connections
Background Jobs Queue quota Job queue with per-tenant limits 100 jobs/min

Quota Enforcement Architecture

graph TB
    Request[User Request
tenant_id = 'A']

    subgraph "API Gateway"
        Auth[Authentication
Extract tenant_id]
        RL[Rate Limiter
Check quota]
        Cache[Quota Cache
Redis]
    end

    subgraph "Application"
        App[App Server]
        Meter[Resource Metering
Track usage]
    end

    subgraph "Database"
        DB[(Database)]
        Limiter[Query Resource Governor
CPU/IOPS limits]
    end

    Request --> Auth
    Auth --> RL
    RL --> Cache
    Cache --> |Over quota| Reject[Return 429 Too Many Requests]
    Cache --> |Within quota| App

    App --> Meter
    Meter --> |Update usage| Cache
    App --> DB
    DB --> Limiter

Rate Limiting Algorithms

1. Token Bucket

  • How: Bucket holds N tokens. Each request consumes 1 token. Refill at rate R/sec.
  • Pros: Allows bursts (if bucket full). Smooth rate limiting.
  • Cons: Need to track bucket state per tenant.
  • Example: 1000 token bucket, refill 100/sec = allow burst of 1000, then steady 100/sec.

2. Leaky Bucket

  • How: Requests enter queue. Process at fixed rate R/sec.
  • Pros: Perfectly smooth output rate.
  • Cons: Queue can grow large. Adds latency.

3. Fixed Window

  • How: Count requests in time window (e.g., 00:00-00:59). Reset each window.
  • Pros: Simple to implement.
  • Cons: Burst at window boundary (1000 at 00:59, 1000 at 01:00 = 2000 in 1 sec).

4. Sliding Window

  • How: Count requests in last N seconds (rolling window).
  • Pros: No boundary burst issue. Fair.
  • Cons: More complex (track timestamps).

Database Resource Quotas

PostgreSQL: Resource Quotas Extension

-- Limit CPU time per query
ALTER ROLE tenant_a SET statement_timeout = '30s';

-- Limit memory per query
ALTER ROLE tenant_a SET work_mem = '256MB';

-- Limit connections
ALTER ROLE tenant_a CONNECTION LIMIT 50;

MySQL: Resource Groups

-- Create resource group for tenant
CREATE RESOURCE GROUP tenant_a_group
  TYPE = USER
  VCPU = 2  -- Max 2 CPUs
  THREAD_PRIORITY = 0;

MongoDB: WiredTiger Cache Limit

// Limit cache per connection (client-side)
mongod --wiredTigerCacheSizeGB 4

Soft vs Hard Limits

Type Behavior Example Use Case
Hard Limit Immediate rejection when exceeded API returns 429 after 1000 req/min Prevent abuse, protect system
Soft Limit Warning or throttling, but allow Slow down queries after 80% quota Encourage upgrade, avoid sudden failure
Burstable Allow temporary burst above limit 1000 req/min avg, burst to 2000 Handle traffic spikes gracefully

Quota Monitoring and Alerts

graph LR
    Meter["Resource Metering Per-tenant metrics"]

        W1[Warning: 80% quota]
        W2[Critical: 95% quota]
        W3[Limit: 100% quota]

        A2[Throttle: Slow down requests]
        A3[Block: Return 429 errors]

    Meter --> W1
    Meter --> W2
    Meter --> W3

    W1 --> A1
    W2 --> A2
    W3 --> A3
Interview Tip: Explain quotas as the primary defense against noisy neighbors. Discuss the trade-off: strict quotas = fair but may frustrate legitimate users; loose quotas = flexible but risk noisy neighbors. Mention tiered quotas (free = 100 req/min, premium = 10K req/min). Discuss token bucket algorithm for rate limiting. Always combine with monitoring and graduated alerts (80% warning, 100% block).

Data Isolation

Data isolation ensures tenants cannot access each other's data, even if there's a bug in application logic. It's the most critical security requirement in multi-tenant systems.

Isolation Techniques

1. Application-Level Filtering (Weakest)

  • Method: Every query includes WHERE tenant_id = ?
  • Risk: Developer forgets filter → data leak
  • Mitigation: ORM-level enforcement, code review, audits
// Dangerous: Missing WHERE clause
SELECT * FROM orders;  // Returns all tenants' orders!

// Safe: Explicit filter
SELECT * FROM orders WHERE tenant_id = 'tenant-123';

2. Row-Level Security (Better)

  • Method: Database enforces isolation at row level
  • Benefit: Impossible to forget filter (database blocks it)
  • Support: PostgreSQL RLS, Oracle VPD, SQL Server RLS
-- PostgreSQL Row-Level Security
CREATE POLICY tenant_isolation ON orders
  USING (tenant_id = current_setting('app.tenant_id')::text);

ALTER TABLE orders ENABLE ROW LEVEL SECURITY;

-- Set tenant context per session
SET app.tenant_id = 'tenant-123';

-- Query automatically filtered by RLS policy
SELECT * FROM orders;  // Only returns tenant-123's orders

3. Schema/Database Separation (Strong)

  • Method: Each tenant has separate schema or database
  • Benefit: Physical isolation (cross-tenant query fails)
  • Example: SELECT * FROM tenant_a.orders vs tenant_b.orders

4. Encryption per Tenant (Strongest)

  • Method: Encrypt each tenant's data with separate keys
  • Benefit: Even with direct DB access, can't read other tenants
  • Use: Highly regulated industries (HIPAA, GDPR)

Row-Level Security Deep Dive

graph TB
    App[Application]
    Conn[Database Connection
Set tenant_id = 'A']

    subgraph "PostgreSQL with RLS"
        RLS[Row-Level Security Policy
Filter: tenant_id = current_tenant]
        Table[Orders Table
tenant_id, order_id, amount]
    end

    App --> |SET app.tenant_id = 'A'| Conn
    Conn --> |SELECT * FROM orders| RLS
    RLS --> |Automatic filter:
WHERE tenant_id = 'A'| Table
    Table --> |Return only Tenant A rows| Conn

Encryption per Tenant

Architecture

  • Key hierarchy: Master key → Tenant key → Data encryption key
  • Master key: Stored in AWS KMS / Azure Key Vault
  • Tenant keys: Derived from master, unique per tenant
  • DEKs: Encrypt actual data, stored with data (encrypted by tenant key)

Workflow

  1. User uploads document for Tenant A
  2. Application retrieves Tenant A's key from KMS
  3. Encrypt document with Tenant A's key
  4. Store encrypted document + encrypted DEK
  5. Tenant B cannot decrypt (doesn't have Tenant A's key)

Benefits

  • Compliance: Meets data sovereignty requirements
  • Data breach protection: Stolen DB dump is encrypted
  • Key rotation: Rotate keys per tenant independently

Downsides

  • Performance: Encrypt/decrypt overhead (10-30% CPU)
  • Complexity: Key management infrastructure
  • Cost: KMS API calls add up at scale

Data Isolation Testing

1. Automated Tests

  • Create Tenant A and Tenant B
  • Insert data for both
  • Query as Tenant A, assert only Tenant A data returned
  • Try to bypass filter (SQL injection, parameter tampering)

2. Penetration Testing

  • Attempt to access other tenant's data via API
  • Test IDOR (Insecure Direct Object Reference) vulnerabilities
  • Example: Change tenant_id in JWT, try to access data

3. Audit Logging

  • Log all cross-tenant data access attempts
  • Alert on anomalies (Tenant A accessing Tenant B's data)
  • Investigate and fix bugs immediately
Interview Tip: Data isolation is the most important security aspect of multi-tenancy. Explain the defense-in-depth approach: application filtering (weakest), RLS (strong), separate DB (stronger), encryption (strongest). Mention real-world breaches where missing WHERE tenant_id exposed data. Discuss PostgreSQL RLS as the gold standard for shared DB. For highly regulated industries, mention encryption per tenant with separate keys.

Tenant Routing

Tenant routing determines how requests are mapped to the correct tenant's data and resources. The routing strategy depends on the isolation model.

Routing Strategies

1. Tenant ID in Request Header

  • Method: Client sends X-Tenant-ID: acme-corp header
  • Pros: Simple, works with any domain
  • Cons: Client can spoof header (need authentication)
GET /api/orders HTTP/1.1
Host: api.example.com
Authorization: Bearer <jwt-token>
X-Tenant-ID: acme-corp

2. Subdomain-Based Routing

  • Method: Each tenant has subdomain: acme.example.com
  • Pros: Clear tenant identity, can route to different servers
  • Cons: SSL cert management (wildcard or per-tenant certs)
// Tenant A
https://acme.example.com/api/orders

// Tenant B
https://beta.example.com/api/orders

3. Path-Based Routing

  • Method: Tenant ID in URL path: /api/acme-corp/orders
  • Pros: Easy to implement, single domain
  • Cons: Ugly URLs, need to extract tenant from path

4. JWT Token (Embedded Tenant ID)

  • Method: Tenant ID in JWT payload: { "tenant_id": "acme-corp" }
  • Pros: Secure (signed token), no need for separate header
  • Cons: Must validate JWT on every request

Routing Architecture

graph TB
    User[User Request
acme.example.com/api/orders]

    subgraph "Edge Layer"
        LB[Load Balancer
Extract subdomain]
        Router[Tenant Router
Lookup shard]
    end

    subgraph "Routing Table"
        RT[Routing DB
tenant → shard mapping]
    end

    subgraph "Application Shards"
        Shard1[Shard 1
Small tenants]
        Shard2[Shard 2
Medium tenants]
        Shard3[Shard 3
Enterprise - Acme]
    end

    User --> LB
    LB --> |tenant_id = 'acme'| Router
    Router --> RT
    RT --> |acme → Shard 3| Router
    Router --> Shard3

Tenant Lookup and Caching

Problem: Performance

Looking up tenant shard/database for every request adds latency.

Solution: Multi-Level Caching

  • L1 (Application memory): In-memory cache (5-min TTL)
  • L2 (Redis): Shared cache across app servers (1-hour TTL)
  • L3 (Database): Authoritative source

Lookup Flow

  1. Check L1 cache (in-memory): HIT → return in 0.01ms
  2. Check L2 cache (Redis): HIT → return in 1ms
  3. Check L3 (Database): MISS → return in 10ms, cache in L1 and L2

Cache Invalidation

  • When tenant moves shards (rebalancing)
  • Invalidate L1 and L2 caches immediately
  • Use cache versioning or pub/sub for invalidation

Custom Domains

Requirement

Enterprise tenants want custom domains: app.acme.com instead of acme.example.com

Implementation

  • DNS CNAME: Customer creates CNAME pointing to your load balancer
  • Domain verification: Tenant adds TXT record to prove ownership
  • SSL certificate: Auto-generate via Let's Encrypt or AWS ACM
  • Routing: Map app.acme.com → tenant ID in routing table

Challenges

  • SSL cert management: Generate, renew, revoke certs per domain
  • Wildcard limits: Can't wildcard across tenants (need per-domain certs)
  • DNS propagation: Custom domain changes take time (TTL)
Interview Tip: Explain subdomain routing as the most common approach for B2B SaaS (Slack, Salesforce). Mention the tenant lookup caching strategy (L1 in-memory, L2 Redis) to avoid latency. For custom domains, discuss DNS CNAME + Let's Encrypt automation. Explain JWT-based routing for stateless API authentication where tenant ID is embedded in the token.

Scaling Multi-Tenant Systems

Scaling multi-tenant systems requires balancing tenant distribution across infrastructure while maintaining isolation and performance.

Tenant Placement Strategies

1. Bin Packing (Cost-Optimized)

  • Goal: Maximize tenants per server (minimize cost)
  • Algorithm: Pack small tenants together, isolate large tenants
  • Risk: Noisy neighbor if estimation is wrong

2. Consistent Hashing

  • Goal: Distribute tenants evenly, minimize reassignment on scale
  • Algorithm: Hash(tenant_id) % N shards
  • Benefit: Add shard → only 1/N tenants move

3. Tiered Placement

  • Tiers: Free (shared), Pro (shared), Enterprise (dedicated)
  • Free tier: 1000 tenants per shard, no guarantees
  • Pro tier: 100 tenants per shard, quota guarantees
  • Enterprise tier: Dedicated infrastructure

4. Geographic Placement

  • Goal: Data residency compliance (GDPR, data sovereignty)
  • Strategy: EU tenants → EU region, US tenants → US region
  • Trade-off: More infrastructure to manage per region

Tenant Sharding Architecture

graph TB
    subgraph "Shard 1 - Small Tenants"
        S1[1000 small tenants
avg 10 req/sec each]
        DB1[(Database 1)]
    end

    subgraph "Shard 2 - Medium Tenants"
        S2[100 medium tenants
avg 100 req/sec each]
        DB2[(Database 2)]
    end

    subgraph "Shard 3 - Enterprise Tenant A"
        S3[Tenant A
10,000 req/sec]
        DB3[(Database 3
Dedicated)]
    end

    subgraph "Shard 4 - Enterprise Tenant B"
        S4[Tenant B
5,000 req/sec]
        DB4[(Database 4
Dedicated)]
    end

    Router[Tenant Router] --> S1
    Router --> S2
    Router --> S3
    Router --> S4

Rebalancing Tenants

When to Rebalance

  • Shard overloaded: CPU > 80%, move tenants to underutilized shard
  • Tenant outgrows tier: Small tenant becomes large, move to dedicated shard
  • New shards added: Redistribute for even load

Rebalancing Process

  1. Identify target tenant: Tenant to move from Shard A → Shard B
  2. Copy data: Replicate tenant's data to Shard B
  3. Dual-write phase: Write to both shards during transition
  4. Update routing: Point new requests to Shard B
  5. Drain old shard: Wait for in-flight requests to complete
  6. Delete old data: Remove tenant from Shard A

Zero-Downtime Migration

  • Use feature flags to route traffic during migration
  • Dual-write phase ensures no data loss
  • Rollback capability if issues detected

Auto-Scaling Multi-Tenant Systems

Component Scaling Strategy Trigger
Application Servers Horizontal (add more servers) CPU > 70% or request queue depth
Databases Read replicas (scale reads) Read IOPS > 80%
Cache (Redis) Add cluster nodes Memory > 80%, eviction rate high
Background Workers Add worker instances Queue depth > 1000 jobs
Shards Add new shards, rebalance All shards > 70% capacity

Hot Tenant Problem

Problem

One tenant suddenly goes viral or runs a major campaign. Traffic spikes 100x.

Solutions

  • Detect hot tenant: Monitor per-tenant request rate
  • Migrate to dedicated shard: Move hot tenant off shared shard
  • Scale tenant's resources: Add dedicated servers/DB for that tenant
  • Rate limiting: Enforce quota to prevent overload
  • CDN/caching: Cache hot tenant's content at edge

Example: Shopify Black Friday

  • Major merchants see 50-100x traffic spike
  • Shopify pre-provisions dedicated infrastructure for top merchants
  • Auto-scaling adds capacity for medium merchants
  • Small merchants stay on shared infrastructure with burst capacity
Interview Tip: Explain tiered placement (free/pro/enterprise on different shards) as the most practical approach. Discuss the tenant rebalancing process with zero downtime (dual-write phase, gradual cutover). Mention hot tenant problem and detection via per-tenant metrics. Explain consistent hashing for even distribution and minimal reassignment when adding shards.

Cost Attribution and Billing

Multi-tenant systems must track resource usage per tenant for billing, cost analysis, and identifying optimization opportunities.

What to Meter

Resource Metric Granularity Billing Model
API Requests Request count Per request $0.01 per 1K requests
Storage GB-months Daily snapshot $0.10 per GB per month
Compute CPU-hours Per minute $0.05 per vCPU-hour
Bandwidth GB egress Per request $0.05 per GB
Database Queries Query count or IOPS Per query $0.01 per 10K queries
Background Jobs Job count or duration Per job $0.001 per job

Metering Architecture

graph TB
    subgraph "Application"
        API[API Request]
        Meter[Metering Agent
Capture: tenant_id, resource, quantity]
    end

    subgraph "Metering Pipeline"
        Kafka[Kafka
Metering events stream]
        Aggregate[Stream Processor
Aggregate per tenant per hour]
    end

    subgraph "Storage"
        TSDB[(Time-Series DB
InfluxDB, Prometheus)]
        Billing[(Billing DB
Usage records)]
    end

    subgraph "Billing System"
        Invoice[Invoice Generator
Monthly bills]
        Dashboard[Usage Dashboard
Tenant view]
    end

    API --> Meter
    Meter --> Kafka
    Kafka --> Aggregate
    Aggregate --> TSDB
    Aggregate --> Billing
    Billing --> Invoice
    TSDB --> Dashboard

Metering Best Practices

1. Asynchronous Metering

  • Don't block requests waiting for metering
  • Emit metering events to queue (Kafka, Kinesis)
  • Process asynchronously in background

2. Idempotency

  • Use unique event IDs to deduplicate
  • Prevent double-counting if event replayed
  • Critical for accurate billing

3. Aggregation Levels

  • Real-time: Track current usage (rate limiting)
  • Hourly: Aggregate for dashboards
  • Daily: Aggregate for billing
  • Monthly: Roll up for invoice

4. Data Retention

  • Raw events: 30 days (for debugging disputes)
  • Hourly aggregates: 1 year
  • Daily aggregates: Forever (tax/audit requirements)

Cost Attribution Challenges

1. Shared Resources

Problem: Database CPU shared by multiple tenants. How to attribute?

Solutions:

  • Track query execution time per tenant
  • Estimate: (tenant query time) / (total query time) × (DB cost)
  • Or: Use separate DB per tenant (easier attribution but higher cost)

2. Overhead Costs

Problem: Load balancers, monitoring, ops team aren't per-tenant.

Solutions:

  • Allocate overhead proportionally to tenant usage
  • Example: Tenant uses 10% of API requests → allocate 10% of load balancer cost

3. Free Tier / Trial Users

Problem: Free users consume resources but don't pay.

Solutions:

  • Track cost per free user (visibility)
  • Set aggressive quotas to limit free tier costs
  • Monitor conversion rate (free → paid) to justify CAC

Real-World Example: AWS Cost Attribution

Multi-Tenant SaaS on AWS

  • EC2 instances: Tag with tenant_id for dedicated instances, allocate shared instances by CPU usage
  • RDS: Dedicated DB = direct cost, shared DB = allocate by query time
  • S3 storage: Use bucket prefixes (s3://bucket/tenant-123/) for per-tenant metering
  • Lambda: Tag invocations with tenant_id, AWS Cost Allocation Tags
  • CloudWatch: Per-tenant metrics via custom dimensions

AWS Cost and Usage Report (CUR)

  • Download detailed billing data
  • Parse and attribute to tenants based on tags
  • Build custom dashboards for cost visibility

Usage-Based Pricing Models

Model Description Example
Tiered Fixed tiers with usage limits Free (1K req/mo), Pro ($50 for 100K), Enterprise (custom)
Pay-As-You-Go Price per unit of usage $0.01 per 1K API requests
Hybrid Base fee + overage charges $100/mo includes 50K requests, then $0.02 per 1K
Commitment Prepay for capacity, discount Reserve 1M requests/mo for $500 (vs $1000 pay-as-you-go)
Interview Tip: Explain the importance of metering for both billing and identifying cost optimization opportunities. Discuss the challenge of shared resource attribution (e.g., database CPU) and practical solutions (track query time per tenant). Mention asynchronous metering to avoid request latency. Discuss idempotency for accurate billing (critical for financial accuracy). Real-world example: AWS tagging strategy for cost allocation.

Key Takeaways

  • Isolation models range from shared everything (lowest cost, weakest isolation) to separate infrastructure (highest cost, strongest isolation). Most systems use tiered placement.
  • Noisy neighbor problem occurs when one tenant's resource usage impacts others. Mitigate with resource quotas, rate limiting, and tenant sharding.
  • Resource quotas enforce fair sharing via API rate limits, CPU limits (cgroups), memory limits, and IOPS throttling. Use token bucket algorithm for rate limiting.
  • Data isolation requires defense-in-depth: application-level filtering (weakest), row-level security (strong), separate schemas/databases (stronger), encryption per tenant (strongest).
  • Tenant routing uses subdomains, headers, or JWT tokens to identify tenants. Cache routing lookups (L1 in-memory, L2 Redis) for performance.
  • Scaling requires smart tenant placement (bin packing, tiered, geographic) and rebalancing with zero downtime (dual-write phase, gradual cutover).
  • Cost attribution tracks per-tenant resource usage for billing. Meter asynchronously, ensure idempotency, and allocate shared costs proportionally.