Day 14: Integration Tests

Phase 7 · Oct 6, 2026

Agenda (2–3 hours)

• Implement (90 min): Write integration tests for the full system — submit/process/complete lifecycle, idempotency, retry behavior
• Test (60 min): Run tests against local DynamoDB + ElasticMQ (SQS-compatible); verify all scenarios pass
• Review (30 min): Identify any missing test scenarios; add them before moving to Week 3

Test Harness

struct TestHarness {
    db:    DynamoClient,   // DynamoDB Local
    sqs:   SqsClient,      // ElasticMQ
    api:   TaskQueueClient, // tonic client to the running server
    worker: JoinHandle<()>, // background worker task
}

impl TestHarness {
    async fn new() -> Self {
        let db  = dynamo_local_client().await;
        let sqs = elasticmq_client().await;
        create_table(&db).await.unwrap();
        create_queue(&sqs).await.unwrap();
        let worker = tokio::spawn(run_worker(db.clone(), sqs.clone(), cancel.clone()));
        // start API server on random port
        let api = connect_api_client("localhost:0").await;
        TestHarness { db, sqs, api, worker }
    }
}

Test Scenarios

#[tokio::test]
async fn test_submit_and_complete() {
    let h = TestHarness::new().await;
    let resp = h.api.submit_task(SubmitTaskRequest {
        idempotency_key: "idem-1".to_string(),
        task_type: "noop".to_string(),
        ..Default::default()
    }).await.unwrap();
    assert_eq!(resp.status, TaskStatus::Pending);

    // wait for worker to process
    wait_for_status(&h.db, &resp.task_id, TaskStatus::Completed, Duration::from_secs(10)).await;

    // verify event history
    let events = h.db.get_task_events(&resp.task_id).await.unwrap();
    assert_eq!(events.len(), 3); // SUBMITTED, PROCESSING, COMPLETED
}

#[tokio::test]
async fn test_idempotent_submit() {
    let h = TestHarness::new().await;
    let req = make_submit_req("idem-2");
    let r1 = h.api.submit_task(req.clone()).await.unwrap();
    let r2 = h.api.submit_task(req).await.unwrap();
    assert_eq!(r1.task_id, r2.task_id);  // same task returned
}

Additional Scenarios

#[tokio::test]
async fn test_retry_and_dlq() {
    // Task type "fail_always" returns Err for first 5 attempts
    let h = TestHarness::new().await;
    let resp = h.api.submit_task(make_submit_req_type("fail_always", "idem-3")).await.unwrap();

    wait_for_status(&h.db, &resp.task_id, TaskStatus::Dead, Duration::from_secs(60)).await;

    let events = h.db.get_task_events(&resp.task_id).await.unwrap();
    let retry_events = events.iter().filter(|e| matches!(e, TaskEvent::Failed { .. })).count();
    assert_eq!(retry_events, 5);
}

Key Takeaways

• ElasticMQ is an SQS-compatible local server — integration tests don't need a real AWS account
• wait_for_status with a timeout surfaces asynchronous bugs (worker not processing, wrong state transition)
• Test the idempotency key deduplification — this is the most business-critical correctness property
• Integration tests that test the full lifecycle are more valuable than unit tests for this kind of system

Next week: load testing, failure injection, CI/CD, and production readiness review.