Day 13: Circuit Breakers in Tower

Phase 2 · Jun 22, 2026

Agenda (2–3 hours)

• Read (45 min): Nygard "Release It!" Chapter 5 (circuit breaker pattern); failsafe-rs README; Martin Fowler's circuit breaker post
• Study (45 min): Model the three states as a Rust enum; identify all state transitions and the events that trigger them
• Practice (45 min): Implement a circuit breaker Layer as a Tower middleware; test all state transitions
• Challenge (30 min): How would you implement a half-open probing strategy that allows exactly 1 probe request at a time (not N)?

Circuit Breaker States

                 failure threshold reached
[Closed] ───────────────────────────────────→ [Open]
   ↑                                              │
   │  probe succeeds                              │ after timeout
   │                                              ↓
[Half-Open] ←──────────────────────────── [Open → Half-Open]

• Closed: normal operation; failures are counted
• Open: fast-fail all requests; protect downstream from overload
• Half-Open: allow one probe request; if it succeeds → Closed; if it fails → Open

State Machine in Rust

#[derive(Clone)]
enum State {
    Closed { failure_count: u32 },
    Open { until: Instant },
    HalfOpen,
}

impl State {
    fn record_failure(&self, threshold: u32) -> State {
        match self {
            State::Closed { failure_count } if failure_count + 1 >= threshold =>
                State::Open { until: Instant::now() + OPEN_DURATION },
            State::Closed { failure_count } =>
                State::Closed { failure_count: failure_count + 1 },
            State::HalfOpen =>
                State::Open { until: Instant::now() + OPEN_DURATION },
            other => other.clone(),
        }
    }
}

CircuitBreaker as Tower Service

impl<S, Req> Service<Req> for CircuitBreaker<S>
where S: Service<Req> {
    fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        let state = self.state.read().unwrap();
        match *state {
            State::Open { until } if Instant::now() < until => {
                // Fast-fail: don't even call inner
                Poll::Ready(Err(CircuitBreakerError::Open))
            }
            _ => self.inner.poll_ready(cx).map_err(CircuitBreakerError::Service)
        }
    }

    fn call(&mut self, req: Req) -> Self::Future {
        // Track success/failure; update state in the response future
        CircuitFuture { inner: self.inner.call(req), state: self.state.clone() }
    }
}

Production Considerations

Failure classification: not all errors should trip the breaker

• Network timeouts: yes
• 404 Not Found: usually no
• 500 Internal Server Error: yes
• Invalid request (400): no

Per-backend circuit breakers: in a load-balanced pool, each backend gets its own circuit breaker so a single bad node doesn't open the circuit for all traffic.

Metrics: expose breaker state and transition counts; alert on frequent Open transitions.

Key Takeaways

• Circuit breaker prevents cascading failure by fast-failing when downstream is degraded
• Three states: Closed (counting), Open (fast-fail), Half-Open (probing)
• Implement as a Tower Layer with shared state (RwLock or atomic) across clones
• Classify errors carefully: only transient/overload errors should trip the breaker
• Combined with retry (inside the breaker): retries won't pile up when circuit is open

Tomorrow: Challenge — layered middleware stack with retry + circuit breaker.