Day 18: Outbox Pattern and Change Data Capture

Phase 4 · Aug 8, 2026

Agenda (2–3 hours)

• Read (45 min): Microservices.io outbox pattern; Debezium CDC documentation
• Study (45 min): What is the "dual write" problem? Why does writing to the DB AND publishing to a message broker in the same request violate consistency?
• Practice (45 min): Implement the outbox pattern: write the event to the same DB transaction as the business entity; add a relay that polls the outbox and publishes to SQS
• Challenge (30 min): How does Debezium use PostgreSQL WAL for CDC? What is the difference between a polling relay and a CDC-based relay?

The Dual Write Problem

Naive approach: save entity to DB, then publish event to message broker.

1. UPDATE orders SET status = 'shipped'
2. publish("order-shipped", event)

What if step 2 fails? Order is marked shipped in DB but no event was published. State inconsistency.

What if we reverse? Event published but DB write fails → event about a change that didn't happen.

No distributed transaction between DB and message broker (without 2PC/XA, which is expensive and complex).

Transactional Outbox

Write the event in the same database transaction as the business entity change:

BEGIN;
UPDATE orders SET status = 'shipped' WHERE id = $1;
INSERT INTO outbox (aggregate_id, event_type, payload, created_at)
VALUES ($1, 'OrderShipped', $2, now());
COMMIT;

If either fails, both roll back. The outbox is now the reliable staging area for events.

Outbox Relay

A background process reads from the outbox and publishes to the message broker:

async fn relay_loop(db: &PgPool, sqs: &SqsClient, queue_url: &str) {
    loop {
        let events = sqlx::query_as!(OutboxEntry,
            "SELECT * FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100"
        ).fetch_all(db).await.unwrap();

        for event in events {
            sqs.send_message()
                .queue_url(queue_url)
                .message_body(&event.payload)
                .message_deduplication_id(&event.id.to_string()) // FIFO queue dedup
                .send().await.unwrap();

            sqlx::query!("UPDATE outbox SET published_at = now() WHERE id = $1", event.id)
                .execute(db).await.unwrap();
        }
        sleep(Duration::from_millis(100)).await;
    }
}

Change Data Capture (CDC)

Instead of polling, use the database's replication log (WAL) to detect changes:

Debezium streams PostgreSQL WAL changes as Kafka messages:

• Zero latency (event published as soon as WAL record is written)
• No polling overhead
• Handles DB restarts gracefully
• Schema changes are tracked

CDC is strictly better than polling for high-volume event-driven architectures.

Key Takeaways

• Dual write is unsafe: use the transactional outbox to atomically record events
• Outbox relay: poll or use CDC (Debezium) to publish outbox entries to the broker
• SQS FIFO queue MessageDeduplicationId = outbox entry ID ensures exactly-once delivery
• CDC (WAL-based) is superior to polling for latency and efficiency at scale

Tomorrow: event-driven architecture — choreography, dead-letter queues, and Kafka.