Day 5: Message Ordering and Deduplication

Phase 5 · Aug 16, 2026

Agenda (2–3 hours)

• Read (45 min): SQS FIFO queue documentation; SQS message group IDs and deduplication IDs documentation
• Study (45 min): How does MessageGroupId enable parallelism within a FIFO queue? Design a group ID strategy for a multi-tenant order processing system
• Practice (45 min): Produce 100 messages to a FIFO queue across 10 groups; verify ordering within each group with two concurrent consumers
• Challenge (30 min): A message with MessageDeduplicationId = X is sent twice within 5 minutes. The second send returns success but the message is not delivered. How should your producer handle this?

FIFO Queue Mechanics

FIFO (First-In, First-Out) queues guarantee:

1. Strict ordering within a MessageGroupId
2. Exactly-once delivery within a 5-minute deduplication window

A message group is like a virtual lane:

• Messages within a group are processed in order
• Different groups are processed in parallel (different consumers)

MessageGroupId Design

Single group ("all"):
  - Total throughput: 300 msg/s
  - All messages strictly ordered
  - Good for: low-volume, global ordering required

Per-customer group (order_id or customer_id):
  - Total throughput: 300 × num_active_groups msg/s
  - Ordering per customer, parallelism across customers
  - Good for: multi-tenant order processing

Per-entity group:
  - Maximum parallelism
  - Ordering per entity
  - Good for: event sourcing, per-aggregate ordering

MessageDeduplicationId

Prevents duplicate delivery within a 5-minute window:

// Option 1: Content-based deduplication (auto SHA256 of body)
// Enable: ContentBasedDeduplication = true on queue

// Option 2: Explicit deduplication ID
sqs.send_message()
    .queue_url(fifo_queue_url)
    .message_body(serde_json::to_string(&order)?)
    .message_group_id(order.customer_id.to_string())
    .message_deduplication_id(order.idempotency_key.to_string())
    .send().await?;

If a message with the same deduplication ID was sent in the last 5 minutes, this send is silently dropped (and returns success). The producer should treat this as a successful send.

Ordering Across Multiple Queues

SQS FIFO guarantees ordering within one queue. For cross-queue ordering:

1. EventBridge FIFO → multiple FIFO SQS queues: ordering preserved end-to-end
2. Sequence numbers in message body: consumers enforce ordering based on application-level sequence (reject/requeue out-of-order messages)
3. Accept eventual ordering: most business workflows tolerate slight reordering across queues

Key Takeaways

• MessageGroupId defines ordering scope; messages in the same group are strictly ordered
• Multiple groups allow parallelism (different consumers) while maintaining per-group ordering
• MessageDeduplicationId prevents duplicate delivery within 5 minutes
• Content-based deduplication (SHA256 of body) works when message content is deterministic

Tomorrow: Lambda integration with SQS — serverless consumer patterns.