Chapter 45: Cost Optimization at Scale
Why Cost Matters
- Super-linear growth: Cloud bills grow faster than traffic—cross-region replication, data transfer, multi-AZ deployments compound costs
- Operational leverage: 10% cost reduction = 10% margin improvement without changing revenue
- FinOps culture: Engineers must own cost—every architectural decision has a monthly bill attached
Interview: "Cost optimization isn't an afterthought—at scale, a poorly chosen data transfer pattern can cost millions annually."
Compute Pricing Models
| Model | Commitment | Savings vs On-Demand | Flexibility | Best For |
|---|---|---|---|---|
| On-Demand | None | 0% (baseline) | Full—start/stop anytime | Spiky workloads, dev/test |
| Reserved (1yr) | 1 year | 40-45% | Regional/AZ-specific | Steady baseline capacity |
| Reserved (3yr) | 3 years | 60-72% | Locked family/region | Proven long-term workloads |
| Spot | None | 60-90% | Can be interrupted (2min notice) | Fault-tolerant batch, stateless workers |
| Savings Plans | 1-3 years | 40-66% | Flexible instance family | Variable workloads within commitment |
Strategy: Reserve instances for baseline (24/7 core services), on-demand for peak overflow, spot for batch processing.
Spot Instance Architecture
| Pattern | Technique | Recovery Time |
|---|---|---|
| Stateless workers | Pull from queue, process, ack—interruption = work requeued | 0s (work automatically retried) |
| Checkpointing | Persist progress every 30-60s, resume on new instance | <1min |
| Multi-AZ diversification | Spot pools across 3+ AZs and instance families | Reduces interruption probability |
| Fallback to on-demand | Auto Scaling launches on-demand if spot unavailable | 2-5min |
Interview: "We run ML training on spot with checkpointing every 5 minutes. Interruptions just resume from last checkpoint—saved 80% on compute costs."
S3 Storage Tier Pricing
| Tier | $/GB/Month | Retrieval Cost | Min Duration | Use Case |
|---|---|---|---|---|
| S3 Standard | $0.023 | None | None | Frequent access (>1/month) |
| S3 Intelligent-Tiering | $0.023 + $0.0025 monitoring | Auto-optimizes | None | Unknown/changing access patterns |
| S3 Standard-IA | $0.0125 | $0.01/GB | 30 days | Infrequent access (quarterly backups) |
| S3 One Zone-IA | $0.01 | $0.01/GB | 30 days | Reproducible data, non-critical |
| S3 Glacier Instant | $0.004 | $0.03/GB | 90 days | Archive with instant retrieval |
| S3 Glacier Flexible | $0.0036 | $0.02/GB (5-12hr) | 90 days | Archive (rare access) |
| S3 Glacier Deep Archive | $0.00099 | $0.02/GB (12-48hr) | 180 days | Compliance, 7-10yr retention |
Lifecycle policy example: Standard → Standard-IA (30 days) → Glacier Flexible (90 days) → Deep Archive (1 year).
Data Transfer Costs (AWS)
| Path | Cost | Optimization |
|---|---|---|
| Intra-AZ (same subnet) | Free | Prefer for microservice communication |
| Inter-AZ (same region) | $0.01/GB (each direction) | Minimize cross-AZ queries—cache aggressively |
| Inter-region | $0.02/GB | Use CloudFront or regional replicas |
| Egress to internet | $0.09/GB (first 10TB) | CDN caching, compression, API pagination |
| Ingress | Free | — |
| CloudFront → Origin | Free (same region) | Ensure CF and origin in same region |
Impact: 1TB/day cross-AZ traffic = $600/month. Same traffic via CDN egress = $2,700/month. Caching a 90% hit rate = $270/month.
Interview: "We reduced data transfer costs 70% by moving read replicas into the same AZ as compute and adding Redis for cross-AZ queries."
Caching ROI Math
Formula:
Monthly Savings = Hit Rate × QPS × DB Cost per Query × 2,592,000 (seconds/month) Break-even = Cache Cost / Monthly Savings (in months)
| Scenario | QPS | Hit Rate | DB Cost/Query | Monthly DB Cost Avoided | Cache Cost (Redis) | Net Savings |
|---|---|---|---|---|---|---|
| Low traffic | 10 | 80% | $0.0001 | $207 | $50 (t4g.small) | $157/mo |
| Medium traffic | 500 | 85% | $0.00005 | $5,508 | $200 (r6g.large) | $5,308/mo |
| High traffic | 5,000 | 90% | $0.00003 | $34,992 | $800 (r6g.2xlarge) | $34,192/mo |
Rule of thumb: Cache pays for itself if it saves >10× its monthly cost. Hit rates >70% + QPS >100 almost always justify caching.
Database Pricing Comparison
| Service | Model | Cost (medium workload) | Scaling | Best For |
|---|---|---|---|---|
| RDS MySQL (m5.large) | Instance-based | $140/mo + storage | Vertical (downtime), read replicas | Traditional workloads, predictable load |
| Aurora MySQL | Instance + I/O | $175/mo + $0.20/M I/O | Read replicas (15), auto-scaling storage | High availability, read-heavy |
| Aurora Serverless v2 | ACU (compute units) | $0.12/ACU-hour (min 0.5 ACU) | Auto-scales in seconds | Variable/unpredictable traffic |
| DynamoDB On-Demand | Per-request | $1.25/M reads, $6.25/M writes | Instant, unlimited | Spiky traffic, unknown patterns |
| DynamoDB Provisioned | Capacity units | $0.00065/WCU-hour | Auto-scaling with lag | Steady, predictable traffic |
Break-even: DynamoDB on-demand cheaper than provisioned if traffic varies >3× daily. Aurora Serverless cheaper than provisioned if idle >12hrs/day.
CDN Economics
Cost formula:
CDN Monthly Cost = Traffic × (1 - Cache Hit Rate) × Origin Cost + Traffic × CDN Cost Net Savings = Traffic × Cache Hit Rate × (Egress Cost - CDN Cost)
| Monthly Traffic | Cache Hit Rate | S3 Egress Cost | CloudFront Cost | Savings |
|---|---|---|---|---|
| 10TB | 80% | $900 (no CDN) | $850 ($0.085/GB) | $50/mo (6%) |
| 100TB | 85% | $8,500 | $5,100 ($0.060/GB avg) | $3,400/mo (40%) |
| 1PB | 90% | $87,000 | $40,000 ($0.040/GB avg) | $47,000/mo (54%) |
CDN pays for itself when: Traffic >10TB/month + cache hit rate >70% OR dynamic content with geo-distribution needs.
Serverless vs Containers Break-Even
| Architecture | Cost Model | ~5M Requests/Month | ~50M Requests/Month | Best For |
|---|---|---|---|---|
| Lambda (128MB, 200ms avg) | $0.20/M requests + $0.0000166667/GB-sec | $17 | $170 | Sporadic, event-driven, <5M req/mo |
| Fargate (0.25 vCPU, 0.5GB) | $0.04048/vCPU-hour + $0.004445/GB-hour | $33/mo (24/7) | $33/mo (same instance) | Steady traffic, long-lived connections |
| ECS on EC2 (t4g.small) | Instance cost ÷ containers | $12/mo (reserved) | $12/mo (reserved) | Predictable, high utilization |
Rule of thumb: Lambda cheaper if traffic <5M req/month OR highly variable. Containers cheaper if always-on OR >10M req/month.
Interview: "We moved from Lambda to Fargate Spot when we crossed 8M requests/month—saved 60% while keeping sub-50ms latency."
Cost-Efficient Architecture Patterns
- Right-sizing: Monitor CPU/memory utilization—downsize if <40% avg, upsize if >80% p95. Use auto-scaling for traffic variance.
- Queue-based leveling: Buffer spikes with SQS/Kafka—process with steady, right-sized compute instead of over-provisioning for peak.
- Compression: gzip/brotli reduces egress costs 60-80%. Compress API responses, logs, backups.
- Request collapsing: Dedupe simultaneous identical requests (cache stampede protection)—saves backend load and DB queries.
- Tiered storage: Hot data in memory (Redis), warm in SSD (RDS), cold in S3 Glacier. Auto-expire or archive unused data.
- Batch processing: Aggregate writes (batch DynamoDB puts), schedule non-urgent jobs during off-peak for spot pricing.
- Multi-tenancy: Share infrastructure across customers with logical isolation—reduces per-tenant overhead.
- Data locality: Colocate compute and storage in same AZ/region—eliminates cross-AZ transfer fees.
FinOps Operational Practices
- Tagging strategy: Enforce tags for team, environment, cost-center, service. Use tag policies to auto-allocate costs.
- Budget alerts: Set budgets per team/service with 80%/100%/120% alerts. Escalate anomalies to owners within 24hrs.
- Right-sizing reviews: Quarterly review of instance utilization. Downsize <40% avg CPU, consolidate underutilized services.
- Waste detection: Identify idle resources (stopped instances still paying for storage, unattached EBS volumes, old snapshots), unused RIs.
- Commitment optimization: Analyze 6-month trends—purchase RIs/Savings Plans for stable baseline, leave spikes on-demand.
- Showback/chargeback: Monthly cost reports per team. Incentivize ownership—teams optimize what they pay for.
- Cost-aware CI/CD: Terminate dev environments after hours, use smaller instances for test, auto-delete PR environments after merge.
Interview: "We implemented tagging and monthly showback reports—teams reduced costs 25% in 6 months just by seeing their spend and optimizing obvious waste."
Key Takeaways
- Cost optimization is architecture—choose compute models, storage tiers, and data paths based on access patterns and scale
- Reserve baseline capacity (RIs/Savings Plans), use spot for batch, on-demand for spikes
- Data transfer is expensive—cache aggressively, minimize cross-AZ/cross-region traffic, use CDNs for high egress
- FinOps culture: tag everything, alert on anomalies, review quarterly, make teams own their costs