Chapter 17: Object/Blob Storage

Object vs File vs Block Storage

S3 Architecture

Components

• Namespace: Global bucket names, region-scoped data. Flat hierarchy (keys simulate folders).
• Metadata Store: Distributed KV store (key → location mapping). Strong consistency since Dec 2020.
• Data Plane: Storage nodes with erasure coding. 11 9's durability via Reed-Solomon.
• Request Router: Routes PUT/GET to correct storage nodes. Handles rate limiting.

Strong Consistency Model

Since December 2020, S3 provides read-after-write and list consistency for all operations:

• PUT: Immediately visible to subsequent GET/LIST
• DELETE: Immediately reflected (object disappears)
• LIST: Returns current state (no eventual consistency lag)
• COPY: Source reflects all prior writes

Durability (11 Nines)

Erasure Coding (Reed-Solomon)

S3 uses erasure coding to achieve 99.999999999% durability (lose 1 object per 100 billion per year).

• Data Shards: Object split into N chunks
• Parity Shards: M additional chunks generated (error correction)
• Reconstruction: Lose up to M shards, still recoverable
• Typical Config: 8 data + 4 parity (can lose 4 of 12 shards)

Replication vs Erasure Coding

Storage Tiers

Cost Example (1TB for 1 year)

Lifecycle Policies

Automatically transition objects between tiers based on age:

{
  "Rules": [{
    "Id": "Archive-old-data",
    "Status": "Enabled",
    "Transitions": [
      { "Days": 30, "StorageClass": "STANDARD_IA" },
      { "Days": 90, "StorageClass": "GLACIER_IR" },
      { "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
    ],
    "Expiration": { "Days": 2555 }
  }]
}

Performance

Request Rate Limits

• PUT/COPY/POST/DELETE: 3,500 requests/sec per prefix
• GET/HEAD: 5,500 requests/sec per prefix
• Prefix: Key path before last slash (e.g., bucket/images/2024/)
• Scaling: Add prefixes to scale horizontally (10 prefixes = 55K GET/s)

Multipart Upload

Required for objects > 5GB, recommended for > 100MB.

• Part Size: 5MB - 5GB (last part can be < 5MB)
• Max Parts: 10,000 per upload
• Max Object: 5TB
• Benefits: Parallel uploads, resume on failure, upload while creating

# AWS CLI multipart upload
aws s3api create-multipart-upload --bucket my-bucket --key large-file.zip

# Upload parts in parallel
aws s3api upload-part --bucket my-bucket --key large-file.zip \
  --part-number 1 --body part1.dat --upload-id XYZ

# Complete
aws s3api complete-multipart-upload --bucket my-bucket --key large-file.zip \
  --upload-id XYZ --multipart-upload '{"Parts":[{"ETag":"abc","PartNumber":1}]}'

Byte-Range Fetches

Download specific byte ranges in parallel (improves large file downloads).

# Download first 1MB
curl -H "Range: bytes=0-1048575" https://bucket.s3.amazonaws.com/file

# Parallel download (4 threads)
bytes_per_chunk=$((file_size / 4))
for i in {0..3}; do
  start=$((i * bytes_per_chunk))
  end=$(((i+1) * bytes_per_chunk - 1))
  curl -H "Range: bytes=$start-$end" ... > part$i &
done
cat part* > complete_file

Transfer Acceleration

Use CloudFront edge locations to accelerate uploads/downloads (50-500% faster for distant regions).

• Endpoint: bucket.s3-accelerate.amazonaws.com
• Cost: $0.04/GB US-EU, $0.08/GB US-Asia
• Protocol: Upload to edge → AWS backbone → S3

Presigned URLs

Generate temporary URLs that grant time-limited access to private S3 objects without AWS credentials.

How It Works

Generation Example

# Python (boto3)
import boto3
from botocore.client import Config

s3 = boto3.client('s3', config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'private/file.pdf'},
    ExpiresIn=3600  # 1 hour
)

# Upload example
upload_url = s3.generate_presigned_post(
    'my-bucket', 'uploads/${filename}',
    ExpiresIn=3600
)

Security Considerations

Direct Browser Upload

<form action="https://bucket.s3.amazonaws.com" method="post" enctype="multipart/form-data">
  <input type="hidden" name="key" value="uploads/${filename}">
  <input type="hidden" name="AWSAccessKeyId" value="AKIAIOSFODNN7EXAMPLE">
  <input type="hidden" name="policy" value="eyJleHBpcmF0aW9uIjoi...">
  <input type="hidden" name="signature" value="s8Qg9...">
  <input type="file" name="file">
  <input type="submit" value="Upload">
</form>

Event Notifications

Trigger workflows when objects are created, deleted, or restored.

Event Types

• s3:ObjectCreated:* (Put, Post, Copy, CompleteMultipartUpload)
• s3:ObjectRemoved:* (Delete, DeleteMarkerCreated)
• s3:ObjectRestore:* (Post, Completed for Glacier)
• s3:Replication:* (OperationFailedReplication, etc.)
• s3:LifecycleTransition

Destinations

Event-Driven Patterns

Configuration Example

{
  "LambdaFunctionConfigurations": [{
    "Id": "process-images",
    "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123:function:ProcessImage",
    "Events": ["s3:ObjectCreated:*"],
    "Filter": {
      "Key": {
        "FilterRules": [
          {"Name": "prefix", "Value": "uploads/"},
          {"Name": "suffix", "Value": ".jpg"}
        ]
      }
    }
  }]
}

Common Use Cases

• Image Processing: Upload → Lambda resize/watermark → save to processed/
• Data Ingestion: Upload CSV → Lambda parse → load to Redshift
• Video Transcoding: Upload video → SQS → EC2 workers → MediaConvert
• Audit Logging: Any change → SNS → CloudWatch Logs
• Data Lake: Upload → EventBridge → Glue Crawler → Athena

Versioning

Keep multiple variants of an object in the same bucket (protect from accidental deletion/overwrite).

States

• Unversioned (default): No version IDs, overwrite replaces object
• Enabled: Every PUT creates new version ID
• Suspended: Stop creating versions, existing versions remain

Version IDs

Delete Markers

• DELETE without version ID: Creates delete marker (object appears deleted, but versions remain)
• DELETE with version ID: Permanently deletes that version
• GET after delete marker: Returns 404
• LIST versions: Shows all versions + delete markers

Cross-Region Replication (CRR)

Automatically replicate objects to another region (requires versioning enabled on both buckets).

• Use Cases: DR, compliance, low-latency access, aggregate logs
• Scope: All objects, prefix filter, or tag filter
• Replication Time: 15 min average (S3 RTC guarantees 99.99% in 15 min for extra cost)
• Delete Replication: Optional (delete markers can replicate)

{
  "Role": "arn:aws:iam::123:role/s3-replication",
  "Rules": [{
    "Status": "Enabled",
    "Priority": 1,
    "Filter": {"Prefix": "documents/"},
    "Destination": {
      "Bucket": "arn:aws:s3:::backup-bucket",
      "ReplicationTime": {"Status": "Enabled", "Time": {"Minutes": 15}},
      "Metrics": {"Status": "Enabled", "EventThreshold": {"Minutes": 15}}
    }
  }]
}

CDN Integration (CloudFront + S3)

Cache S3 objects at edge locations for low-latency global delivery.

Architecture

Origin Access Control (OAC)

Restrict S3 access to only CloudFront (prevents direct S3 access).

# S3 bucket policy (only allow CloudFront)
{
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "cloudfront.amazonaws.com"},
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-bucket/*",
    "Condition": {
      "StringEquals": {
        "AWS:SourceArn": "arn:aws:cloudfront::123:distribution/EDFDVBD6"
      }
    }
  }]
}

Performance Gains

Caching Strategy

• Default TTL: 24 hours (configure via Cache-Control headers)
• Cache Key: URL path + query string (customize to include headers/cookies)
• Invalidation: Manual (costs $0.005 per path after 1000 free/month) or versioned URLs
• Versioned URLs: Append version to filename (e.g., style.v123.css) → cache forever

# S3 object metadata (set Cache-Control)
aws s3 cp file.jpg s3://bucket/file.jpg \
  --metadata-directive REPLACE \
  --cache-control "max-age=31536000, immutable"

# CloudFront invalidation (bust cache)
aws cloudfront create-invalidation \
  --distribution-id EDFDVBD6 \
  --paths "/images/*"

Alternatives to S3

When to Use Each

• S3: AWS ecosystem, full feature set, mature ecosystem
• Azure Blob: Azure ecosystem, lower storage cost (Hot/Cool tiers)
• GCS: GCP ecosystem, multi-region durability (16 9's)
• Cloudflare R2: High egress volume, CDN integration
• Backblaze B2: Large archives, infrequent access, cost-sensitive
• Wasabi: Active archives, no API/egress charges, predictable cost
• MinIO: On-prem, air-gapped, K8s native, compliance
• Ceph: Unified storage (block+file+object), large on-prem deployments

Design Patterns

1. Data Lake Architecture

Key Patterns:

• Zone Architecture: Raw (immutable) → Processed (cleaned) → Curated (aggregated)
• Partitioning: s3://bucket/year=2024/month=05/day=12/data.parquet
• File Format: Parquet/ORC (columnar, compressed) for analytics
• Lifecycle: Raw (30d Standard → Glacier), Processed (90d Standard → IA)
• Catalog: Glue Data Catalog (schema registry, discoverable)

2. Media Processing Pipeline

Key Patterns:

• Presigned Upload: Direct browser → S3 (save backend bandwidth)
• Event-Driven: S3 → SQS → Lambda (handles spikes, retries)
• Outputs: HLS (streaming), MP4 (download), thumbnails (preview)
• CDN: CloudFront caches outputs (reduce S3 egress cost)
• Storage Tiers: Raw (30d → Glacier), Processed (Standard, long TTL)

3. Backup and Disaster Recovery

Key Patterns:

• Versioning: Protect from accidental delete/overwrite
• CRR: Async replication to DR region (15min average)
• Lifecycle: Primary (Standard 30d → Glacier 90d → Deep Archive 365d)
• Immutability: Object Lock (WORM mode, compliance)
• Testing: Quarterly DR drill (restore from S3DR)

Common Anti-Patterns

Production Deep Dive

S3 Durability Calculation: Reed-Solomon Erasure Coding

S3 achieves 11 nines durability (99.999999999%) using Reed-Solomon (8,4) erasure coding:

Multipart Upload Logic (Pseudocode)

Production-grade multipart upload with retry and idempotency:

# Step 1: Initiate multipart upload
response = s3.initiate_multipart_upload(bucket, key)
upload_id = response.upload_id

# Step 2: Upload parts in parallel (100MB chunks)
part_size = 100 * 1024 * 1024  # 100MB
total_parts = ceil(file_size / part_size)
etag_map = {}  # {part_number: ETag}

def upload_part_with_retry(part_number, data):
    max_retries = 3
    for attempt in range(max_retries):
        try:
            response = s3.upload_part(
                bucket=bucket,
                key=key,
                upload_id=upload_id,
                part_number=part_number,
                body=data
            )
            return response.etag  # Idempotent: same part_number overwrites
        except NetworkError as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)  # Exponential backoff

# Parallel upload (4 threads)
with ThreadPoolExecutor(max_workers=4) as executor:
    futures = []
    for part_num in range(1, total_parts + 1):
        offset = (part_num - 1) * part_size
        data = file.read_chunk(offset, part_size)
        future = executor.submit(upload_part_with_retry, part_num, data)
        futures.append((part_num, future))

    for part_num, future in futures:
        etag_map[part_num] = future.result()

# Step 3: Complete multipart upload
parts = [{"PartNumber": num, "ETag": etag} for num, etag in sorted(etag_map.items())]
s3.complete_multipart_upload(
    bucket=bucket,
    key=key,
    upload_id=upload_id,
    multipart_upload={"Parts": parts}
)

# Error handling: If upload abandoned or fails
# - Retry individual parts only (idempotent)
# - Abort upload: s3.abort_multipart_upload(bucket, key, upload_id)
# - Or set lifecycle rule: auto-delete incomplete uploads after 7 days

S3 Request Rate Optimization: Prefix Partitioning

S3 request rate limits (3500 PUT, 5500 GET per prefix) can be scaled horizontally:

Hash Prefix Strategy:

import hashlib

def get_s3_key(user_id, filename):
    # Add 2-character hash prefix for even distribution
    hash_prefix = hashlib.md5(user_id.encode()).hexdigest()[:2]
    return f"{hash_prefix}/images/{user_id}/{filename}"

# Example: user_id=123 → hash=202cb962ac59 → prefix=20
# Distributes across 256 prefixes (00-ff) → 256x scale

Historical Context: Pre-2018, S3 had 100 TPS per prefix limit. July 2018 update increased to 3500 PUT + 5500 GET, making prefix partitioning less critical but still valuable at extreme scale.

Storage Tier Cost Comparison

Cost Calculation Example (1TB archive, 1 retrieval/year):

• S3 Standard: 1000GB × $0.023 × 12 = $276/year
• S3 IA: 1000GB × $0.0125 × 12 + 1000GB × $0.01 = $150 + $10 = $160/year (42% savings)
• Glacier Instant: 1000GB × $0.004 × 12 + 1000GB × $0.03 = $48 + $30 = $78/year (72% savings)
• Glacier Deep: 1000GB × $0.00099 × 12 + 1000GB × $0.02 = $12 + $20 = $32/year (88% savings)