Chapter 24: Message Queues

Why Message Queues

Message queues enable asynchronous communication between services, decoupling producers from consumers and providing buffering, load leveling, and resilience.

Key Benefits

Before/After Architecture

Core Concepts

Basic Flow

Delivery Guarantees

At-Most-Once: Auto ACK

// Consumer auto-ACKs on receive
channel.basicConsume(queueName, autoAck: true, consumer);
// If processing fails, message lost
processMessage(message); // crash here = lost message

At-Least-Once: Manual ACK

// Consumer ACKs after successful processing
channel.basicConsume(queueName, autoAck: false, consumer);
try {
    processMessage(message);
    channel.basicAck(deliveryTag); // ACK only on success
} catch (Exception e) {
    channel.basicNack(deliveryTag, requeue: true); // retry
}

Exactly-Once: Deduplication + Idempotency

// Deduplication via message ID
String messageId = message.getMessageId();
if (processedIds.contains(messageId)) {
    channel.basicAck(deliveryTag); // already processed
    return;
}

// Idempotent processing with database transaction
db.transaction(() -> {
    processMessage(message);
    processedIds.add(messageId); // persist dedup ID
});
channel.basicAck(deliveryTag);

Comparison

RabbitMQ

RabbitMQ implements AMQP (Advanced Message Queuing Protocol), featuring flexible routing via exchanges and strong delivery guarantees.

Architecture

Exchange Types

Topic Exchange Examples

// Wildcards: * matches one word, # matches zero or more words

Routing Key          Binding Pattern        Match?
logs.error.api       logs.error.*           Yes
logs.error.db.slow   logs.error.*           No
logs.error.db.slow   logs.error.#           Yes
logs.info.api        logs.*.api             Yes
logs.error.api.auth  logs.#                 Yes

Core Features

Queue Types (High Availability)

AWS SQS

Simple Queue Service: fully managed, serverless, highly scalable. Two queue types: Standard (best-effort ordering, unlimited throughput) and FIFO (strict ordering, 3000 msg/sec).

Standard vs FIFO Queues

Key Concepts

Visibility Timeout Flow

Long Polling vs Short Polling

Dead Letter Queue (DLQ)

// Configure DLQ on source queue
{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123:my-dlq",
    "maxReceiveCount": 3  // move to DLQ after 3 delivery attempts
  }
}

Message Ordering

Maintaining message order at scale is challenging. Most systems provide ordering guarantees only within partitions or message groups.

Why Ordering is Hard

Ordering Strategies

SQS FIFO Message Groups

// Order maintained per user, but users processed in parallel
sqs.sendMessage({
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123/orders.fifo',
  MessageBody: JSON.stringify(order),
  MessageGroupId: userId,  // all messages for user ordered
  MessageDeduplicationId: orderId
});

Kafka Partition-Level Ordering

Application-Level Sequence Numbers

// Producer adds sequence number
{
  "userId": 123,
  "action": "update_profile",
  "sequenceNumber": 42,
  "data": {...}
}

// Consumer reorders out-of-order messages
Map<UserId, PriorityQueue<Message>> buffers;
long expectedSeq = getNextExpected(userId);

if (message.seq == expectedSeq) {
  process(message);
  expectedSeq++;
  // process buffered messages now in order
  while (!buffers[userId].isEmpty() &&
         buffers[userId].peek().seq == expectedSeq) {
    process(buffers[userId].poll());
    expectedSeq++;
  }
} else {
  buffers[userId].add(message); // buffer for later
}

Dead Letter Queues

Dead Letter Queues (DLQs) isolate poison messages that repeatedly fail processing, preventing them from blocking the main queue.

DLQ Workflow

Common Causes of DLQ Messages

Retry Strategies

DLQ Monitoring

Reprocessing DLQ Messages

// 1. Fix the bug/issue causing failures
// 2. Move messages from DLQ back to main queue

// SQS: Redrive to source queue (AWS Console or API)
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123:my-dlq \
  --destination-arn arn:aws:sqs:us-east-1:123:my-queue

// RabbitMQ: Shovel plugin or manual script
while (message = dlqChannel.basicGet("my-dlq")) {
  mainChannel.basicPublish("", "my-queue", message.properties, message.body);
  dlqChannel.basicAck(message.deliveryTag);
}

Backpressure

Backpressure occurs when consumers can't keep up with producer rate. Without handling, queues grow unbounded, causing memory issues and increased latency.

Symptoms

Mitigation Strategies

Queue Depth Monitoring

// CloudWatch alarm for SQS queue depth
{
  "AlarmName": "HighQueueDepth",
  "MetricName": "ApproximateNumberOfMessagesVisible",
  "Namespace": "AWS/SQS",
  "Statistic": "Average",
  "Period": 300,
  "EvaluationPeriods": 2,
  "Threshold": 10000,
  "ComparisonOperator": "GreaterThanThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:123:pagerduty"]
}

Auto-Scaling Consumers

Consumer Scaling Example (Kubernetes HPA)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: consumer-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: message-consumer
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: sqs_queue_depth
        selector:
          matchLabels:
            queue_name: my-queue
      target:
        type: AverageValue
        averageValue: "1000"  # scale if >1000 msgs per pod

Patterns

1. Work Queue (Task Distribution)

Use case: Distribute tasks across multiple workers (image resize, report generation). Each message consumed by one worker only.

2. Pub/Sub (Event Broadcasting)

Use case: Broadcast events to multiple subscribers (user signup, order placed). Each subscriber gets copy of message.

3. Request-Reply

Use case: RPC over message queue. Client sends request, server replies to specified queue with correlation ID.

4. Priority Queue

// RabbitMQ priority queue
channel.queueDeclare("tasks", true, false, false,
  Map.of("x-max-priority", 10));

// Publish with priority
channel.basicPublish("", "tasks",
  new AMQP.BasicProperties.Builder().priority(5).build(),
  message);

Use case: Premium users, urgent tasks, critical alerts processed first.

5. Delayed/Scheduled Messages

// SQS delay (up to 15 min)
sqs.sendMessage({
  QueueUrl: queueUrl,
  MessageBody: message,
  DelaySeconds: 300  // 5 minutes
});

// RabbitMQ delayed exchange (plugin required)
channel.exchangeDeclare("delayed", "x-delayed-message", true,
  Map.of("x-delayed-type", "direct"));
channel.basicPublish("delayed", routingKey,
  new AMQP.BasicProperties.Builder()
    .headers(Map.of("x-delay", 300000))  // 5 min in ms
    .build(),
  message);

Use case: Scheduled jobs, retry with backoff, reminder emails.

Pattern Comparison

Comparison: RabbitMQ vs SQS vs Others

When to Choose Each

Queue vs Stream: When to Use What

Message queues (RabbitMQ, SQS) and event streams (Kafka, Kinesis) solve different problems. Choosing wrong one causes pain.

Core Differences

Decision Matrix

Use Case Examples

Hybrid Pattern: Queue + Stream

Pattern: Use Kafka for event distribution and replay, then push tasks to SQS for worker distribution. Best of both: event history + task distribution.

Benchmarks & Real-World Architecture

Performance Benchmarks

Real Architecture Example: Uber-Style Ride System

Scenario: Process ride events (trip start, payment, rating) with fan-out to multiple downstream services.

// RabbitMQ flow
Trip Service → Topic Exchange (ride.events)
                    ↓ (binding: ride.*)
    ┌──────────────┼──────────────┐
    ↓              ↓              ↓
pricing-queue  analytics-queue  notification-queue
    ↓              ↓              ↓
Pricing Worker Analytics Worker Notification Worker
(5 instances)  (3 instances)    (2 instances)

Failure Scenario Timeline

Scenario: Consumer crashes before ACKing message. What happens?

Cost & Throughput Comparison