Day 7: Challenge — Rust Service with Circuit Breaker + Bulkhead

Phase 4 · Jul 28, 2026

Challenge Overview

Build a resilient Axum HTTP proxy service that calls a downstream HTTP API.

Requirements:

1. Proxy GET /api/data to a configurable downstream URL
2. Per-downstream circuit breaker: opens after 5 failures in 30s window
3. Bulkhead: max 50 concurrent downstream calls (semaphore)
4. Timeout: 2s per downstream call
5. Fallback: when circuit is open, return cached last-known-good response
6. Metrics: expose circuit state, bulkhead usage, request counts via /metrics
7. Simulate downstream failures with a mock server that returns 503 on demand

Architecture

GET /api/data
     ↓
[Axum Handler]
     ↓
[Bulkhead: Semaphore(50)]
     ↓
[Circuit Breaker: 5/30s]
     ↓
[Timeout: 2s]
     ↓
[reqwest client → downstream]
     ↓ (on circuit open)
[Fallback Cache]

Key Components

struct AppState {
    downstream_url: String,
    circuit: Arc<CircuitBreaker>,
    bulkhead: Arc<Semaphore>,
    fallback_cache: Arc<RwLock<Option<CachedResponse>>>,
    metrics: Arc<Metrics>,
}

async fn proxy_handler(State(state): State<AppState>) -> impl IntoResponse {
    // Acquire bulkhead permit (or return 503)
    let _permit = state.bulkhead
        .try_acquire()
        .map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;

    // Check circuit breaker
    state.circuit.call(|| fetch_downstream(&state)).await
        .unwrap_or_else(|_| state.fallback_cache.read().unwrap().clone().into_response())
}

Mock Downstream

// GET /control?error_rate=0.5 sets the failure rate
// GET /data returns mock data or a 503 based on the error rate
async fn data_handler(State(rate): State<Arc<AtomicU64>>) -> impl IntoResponse {
    let r = rate.load(Ordering::Relaxed) as f64 / 100.0;
    if rand::random::<f64>() < r {
        StatusCode::SERVICE_UNAVAILABLE
    } else {
        Json(json!({ "data": "ok", "ts": Utc::now().to_rfc3339() })).into_response()
    }
}

Test Scenario

# Start downstream mock on :8081
cargo run --bin mock-downstream &

# Start proxy on :8080
cargo run --bin proxy &

# Normal operation
for i in $(seq 1 20); do curl -s localhost:8080/api/data; echo; done

# Ramp up error rate to 80%
curl "localhost:8081/control?error_rate=80"

# Observe circuit opening
for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/api/data; done
# Should see: 200, 200, 503(fallback), 503(circuit open), ...

# Recover
curl "localhost:8081/control?error_rate=0"
# After 30s: circuit should close and return 200 again

Week 1 Recap

Next week: distributed transactions — coordinating state changes across services.