Day 15: Load Testing

Phase 7 · Oct 7, 2026

Agenda (2–3 hours)

• Implement (60 min): Write a ghz or custom Tokio load generator that submits tasks at a target rate; ramp from 50 to 500 tasks/second
• Test (90 min): Run load test against the staging environment; measure P50/P95/P99 latency, error rate, and DynamoDB WCU consumption
• Review (30 min): Compare observed latency to the P99 < 100ms SLO; identify and fix the bottleneck

Load Generator

// Submit tasks at `rate` req/s using a token bucket
async fn load_test(client: TaskQueueClient, rate: u32, duration: Duration) {
    let interval = Duration::from_nanos(1_000_000_000 / rate as u64);
    let mut ticker = tokio::time::interval(interval);
    let deadline = Instant::now() + duration;
    let mut i: u64 = 0;

    while Instant::now() < deadline {
        ticker.tick().await;
        let client = client.clone();
        tokio::spawn(async move {
            let start = Instant::now();
            let result = client.submit_task(SubmitTaskRequest {
                idempotency_key: format!("load-{i}"),
                task_type: "noop".to_string(),
                ..Default::default()
            }).await;
            histogram!("load_test.latency_ms").record(start.elapsed().as_millis() as f64);
            if result.is_err() { counter!("load_test.errors").increment(1); }
        });
        i += 1;
    }
}

Expected Results at 500 req/s

Bottleneck candidates: DynamoDB throttling, tonic connection limits, ECS CPU saturation.

Profiling Bottlenecks

If P99 > 100ms:

1. Check request_duration_seconds histogram by span — which span is slow?
2. Check db.PutItem span duration — DynamoDB throttling?
3. Check ECS task CPU utilization in CloudWatch — API service saturated?
4. Check sqs.SendMessage duration — SQS slow?

If DynamoDB throttles:

• Switch to on-demand capacity mode (auto-scales without provisioning)
• Or increase provisioned WCU and enable auto-scaling

Key Takeaways

• Load test before declaring the SLO met — paper estimates are not measurements
• The token-bucket pattern in Tokio (interval + spawn) produces accurate open-loop load
• P99 latency bottlenecks are rarely where you expect — trace data from the load test reveals the real culprit
• DynamoDB on-demand capacity absorbs load spikes without pre-planning; provisioned mode is cheaper at steady state

Tomorrow: failure injection — chaos engineering for the task queue.