Day 14: Challenge — Middleware Stack with Retry + Circuit Breaker

Phase 2 · Jun 23, 2026

Challenge Overview

Build a Tower middleware stack that simulates a resilient HTTP client calling an unreliable backend.

Requirements:

1. Mock backend that fails 40% of requests with a transient error
2. Retry layer: up to 3 retries with exponential backoff + jitter
3. Per-attempt timeout: 500ms
4. Circuit breaker: opens after 5 consecutive failures, resets after 10s
5. Metrics: count successes, failures, retries, circuit-open rejections
6. Demonstrate: circuit opens under high failure rate; closes after backend recovers

Stack Configuration

// Inner → outer wrapping order:
// [backend] → [timeout] → [retry] → [circuit_breaker] → [rate_limit] → caller

let service = ServiceBuilder::new()
    .layer(RateLimitLayer::new(50, Duration::from_secs(1)))
    .layer(CircuitBreakerLayer::new(CircuitBreakerConfig {
        failure_threshold: 5,
        open_duration: Duration::from_secs(10),
    }))
    .layer(RetryLayer::new(ExponentialBackoffPolicy::new(3)))
    .layer(TimeoutLayer::new(Duration::from_millis(500)))
    .service(mock_backend);

Mock Backend

struct MockBackend {
    failure_rate: f64,  // 0.0 - 1.0
}

impl Service<Request> for MockBackend {
    fn call(&mut self, _req: Request) -> BoxFuture<Result<Response, Error>> {
        let rate = self.failure_rate;
        Box::pin(async move {
            sleep(Duration::from_millis(50 + rand::random::<u64>() % 200)).await;
            if rand::random::<f64>() < rate {
                Err(Error::Transient("backend overloaded"))
            } else {
                Ok(Response::new(200, "ok"))
            }
        })
    }
}

Test Scenarios

#[tokio::test]
async fn circuit_opens_under_high_failure_rate() {
    let backend = MockBackend { failure_rate: 0.9 };
    // ... build stack ...
    // Send 20 requests
    // Assert: some succeed early; circuit opens; later requests fast-fail
    // Assert: after 10s cool-down, circuit closes and requests succeed again
}

#[tokio::test]
async fn retry_succeeds_on_transient_failures() {
    let backend = MockBackend { failure_rate: 0.3 };
    // 3 retries should recover from ~70% of first attempts failing
    // Assert: >95% of requests eventually succeed
}

Week 2 Recap

Next week: Axum — HTTP APIs and WebSockets built on top of this Tower foundation.