Day 12: Load Balancing with Tower

Phase 2 · Jun 21, 2026

Agenda (2–3 hours)

• Read (45 min): tower::balance module documentation; the p2c (power-of-two-choices) algorithm paper abstract; Nginx load balancing guide
• Study (45 min): Why does p2c outperform round-robin in the presence of variable request latency?
• Practice (45 min): Build a load-balanced client that discovers backends from a watch::Receiver<Vec<SocketAddr>> and distributes requests using p2c
• Challenge (30 min): Implement consistent hash load balancing for session-affinity use cases (same client → same backend)

Load Balancing Algorithms

Power-of-Two-Choices (p2c)

Classic result from Mitzenmacher (1996): picking the better of two random choices dramatically outperforms purely random selection.

• Purely random: O(log n / log log n) max load
• p2c: O(log log n) max load
• Round-robin: O(1) but breaks under variable latency

Tower's balance::p2c samples two backends and picks the one with fewer pending requests (tracked by tower::load::PendingRequests).

Tower Balance

use tower::balance::p2c::Balance;
use tower::discover::ServiceList;

// Create backend services
let backends = vec![
    ServiceBuilder::new().service(BackendService::new("10.0.0.1:8080")),
    ServiceBuilder::new().service(BackendService::new("10.0.0.2:8080")),
    ServiceBuilder::new().service(BackendService::new("10.0.0.3:8080")),
];

// Wrap each with load tracking
let tracked = backends.into_iter()
    .map(|s| tower::load::PendingRequests::new(s, tower::load::CompleteOnResponse))
    .collect::<Vec<_>>();

let balancer = Balance::new(ServiceList::new(tracked));

Dynamic Service Discovery

Real load balancers watch for backend changes. Tower's Discover trait models a stream of service additions/removals:

// Using watch channel for live updates
let (discovery_tx, discovery_rx) = watch::channel(initial_backends);

// When backends change:
discovery_tx.send(updated_backends).unwrap();

// Balance automatically uses updated set

Integrate with AWS Cloud Map, Kubernetes endpoints, or a Consul catalog via a background task that pushes updates.

Key Takeaways

• p2c (power-of-two-choices) balances load nearly as well as optimal, with O(1) coordination
• Tower's balance::p2c + load::PendingRequests is the standard pattern
• Dynamic discovery via watch::Receiver enables backend set updates without restart
• Consistent hashing preserves session affinity at the cost of uneven load under churn

Tomorrow: circuit breakers — fail fast when downstream services are unhealthy.