← Week 2: Distributed Transactions

Day 14: Challenge — Saga with Compensation and Idempotent Handlers

Phase 4 · Aug 4, 2026

← Week 2: Distributed Transactions

Challenge Overview

Implement a complete order-placement saga in Rust with compensation, crash recovery, and idempotency.

Business flow:

  1. Reserve inventory (compensate: release reservation)
  2. Charge payment card (compensate: refund)
  3. Create shipment record (compensate: cancel shipment)
  4. Send confirmation email (no compensation — email is best-effort)

Requirements:

  • Orchestrated saga: explicit state machine persisted to SQLite (via rusqlite)
  • Every step is idempotent: safe to retry
  • Simulate failures: random step failures; orchestrator crash mid-saga
  • Compensations run in reverse order on failure
  • Alert (log) if any compensation fails
← Week 2: Distributed Transactions

Saga State Schema

CREATE TABLE sagas (
    id TEXT PRIMARY KEY,
    state TEXT NOT NULL,  -- pending, running, completed, compensating, compensated, failed
    input JSON NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE saga_steps (
    saga_id TEXT NOT NULL REFERENCES sagas(id),
    step_name TEXT NOT NULL,
    state TEXT NOT NULL,    -- pending, running, completed, failed, compensating, compensated
    output JSON,
    error TEXT,
    PRIMARY KEY (saga_id, step_name)
);
← Week 2: Distributed Transactions

Test Scenarios

#[tokio::test]
async fn full_success_path() {
    let orchestrator = build_orchestrator(/* no failures */);
    let saga_id = orchestrator.run(order_input()).await.unwrap();
    let state = orchestrator.get_saga(saga_id).await.unwrap();
    assert_eq!(state.state, SagaState::Completed);
    assert!(state.steps.iter().all(|s| s.state == StepState::Completed));
}

#[tokio::test]
async fn payment_failure_triggers_compensation() {
    let orchestrator = build_orchestrator(/* payment fails */);
    let saga_id = orchestrator.run(order_input()).await;
    let state = orchestrator.get_saga(saga_id).await.unwrap();
    assert_eq!(state.state, SagaState::Compensated);
    // inventory compensation ran; reservation released
}

#[tokio::test]
async fn crash_recovery_resumes_saga() {
    // Start saga, kill orchestrator mid-way, restart, verify completion
}
← Week 2: Distributed Transactions

Idempotency in Steps

// InventoryService::reserve — idempotent
async fn reserve(&self, saga_id: &str, item_id: &str) -> Result<Reservation> {
    // Use saga_id as idempotency key
    if let Some(r) = self.db.find_reservation_by_idempotency(saga_id).await? {
        return Ok(r); // already done, return existing
    }
    let r = self.db.create_reservation(saga_id, item_id).await?;
    Ok(r)
}
← Week 2: Distributed Transactions

Week 2 Recap

Topic Key insight
2PC Strong atomicity; coordinator crash = blocking
3PC Non-blocking but partition-unsafe
Sagas Compensatable local transactions; at-least-once semantics
ACID/MVCC Isolation levels; write skew requires SERIALIZABLE
Distributed locks Fencing tokens; etcd leases > Redlock for critical locks
Idempotency (key → response) cache transforms at-least-once to effectively-exactly-once

Next week: Event Sourcing & CQRS — append-only logs as the source of truth.