Day 3: Implementing Circuit Breakers in Rust

Phase 4 · Jul 24, 2026

Agenda (2–3 hours)

• Read (45 min): failsafe-rs crate documentation; recloser crate README; review Phase 2 circuit breaker Tower implementation
• Study (45 min): Compare your Phase 2 implementation to failsafe-rs; what did you miss?
• Practice (45 min): Wrap a reqwest HTTP client with a circuit breaker using failsafe-rs; test against a mock server that returns varying error rates
• Challenge (30 min): How do you handle the case where the circuit breaker trips mid-request (the request was already in flight when the breaker opened)?

Using failsafe-rs

use failsafe::{CircuitBreaker, Config, Error};
use failsafe::backoff::exponential;
use failsafe::failure_policy::ConsecutiveFailures;

let cb = Config::new()
    .failure_policy(ConsecutiveFailures::new(5))
    .build();

match cb.call(|| my_operation()) {
    Ok(result) => println!("success: {:?}", result),
    Err(Error::Inner(e)) => println!("operation failed: {}", e),
    Err(Error::Rejected) => println!("circuit open, fast-fail"),
}

Tower-Based Circuit Breaker (Recap)

From Phase 2 Week 2 Day 13, extended with time-window metrics:

struct CircuitBreakerState {
    state: RwLock<State>,
    window: Mutex<VecDeque<(Instant, bool)>>, // (time, success)
    config: Config,
}

impl CircuitBreakerState {
    fn record(&self, success: bool) {
        let now = Instant::now();
        let mut window = self.window.lock().unwrap();
        // Evict old entries
        window.retain(|(t, _)| now.duration_since(*t) < self.config.window_size);
        window.push_back((now, success));

        if !success { self.maybe_open(&window); }
        else if matches!(*self.state.read().unwrap(), State::HalfOpen) {
            *self.state.write().unwrap() = State::Closed;
        }
    }
}

Per-Endpoint Breakers in a Load Balancer

In a pool of 5 backends, one is degraded:

Circuit per endpoint:
  backend-1: CLOSED  (healthy)
  backend-2: CLOSED  (healthy)
  backend-3: OPEN    (degraded → auto-ejected)
  backend-4: CLOSED  (healthy)
  backend-5: CLOSED  (healthy)

The load balancer skips backend-3 and distributes traffic across 4 healthy backends. No client-visible degradation.

Implement: wrap each backend service in a CircuitBreaker; poll_ready() returns Err when open.

Key Takeaways

• failsafe-rs provides a production-quality circuit breaker with pluggable policies
• Per-endpoint circuit breakers in a load balancer isolate single-backend failures
• Mid-flight requests that started before the circuit opened are allowed to complete (don't abort them)
• Always pair circuit breaker with a fallback strategy: static response, cache, or graceful degradation

Tomorrow: Bulkhead pattern — isolate failures within the same service.