Day 6: Instrumenting a Real Service

Phase 6 · Sep 7, 2026

Agenda (2–3 hours)

• Read (45 min): OTel semantic conventions for HTTP, database, messaging spans; tracing-opentelemetry README
• Study (45 min): What attributes should a database span always include? What attributes are optional? Design a span naming convention for your service
• Practice (45 min): Add OTel instrumentation to a full Axum + tonic + DynamoDB service; verify trace attributes follow semantic conventions in Jaeger
• Challenge (30 min): An #[instrument] macro captures all function arguments by default. Design a policy for which arguments to skip (PII, large payloads, secrets)

Semantic Conventions

OTel semantic conventions standardize span attribute names so tools work uniformly:

Axum Middleware Layer

use axum_tracing_opentelemetry::middleware::OtelAxumLayer;

let app = Router::new()
    .route("/tasks", get(list_tasks).post(create_task))
    .layer(OtelAxumLayer::default()); // injects trace for each request

OtelAxumLayer:

• Extracts traceparent from incoming request headers
• Creates a root span with http.method, http.route, http.status_code
• Propagates context to tracing::Span::current() for downstream calls

Database Span Attributes

#[instrument(
    skip(client, key),
    fields(
        db.system = "dynamodb",
        db.operation = "GetItem",
        db.name = %table_name,
        db.dynamodb.table_names = %table_name,
    )
)]
async fn get_item(client: &Client, table_name: &str, key: Key) -> Result<Item, Error> {
    // span automatically records duration and error status
    client.get_item()
        .table_name(table_name)
        // ...
        .send().await
        .map_err(|e| { tracing::error!(error = %e); e.into() })
}

Key Takeaways

• OTel semantic conventions ensure consistent span attributes across languages and tools
• axum-tracing-opentelemetry provides HTTP request spans with standard attributes
• Always record db.operation and db.name on database spans; avoid logging query values (may contain PII)
• Use skip(...) in #[instrument] to exclude arguments that are large, secret, or PII

Tomorrow: Phase 6 Week 1 Challenge — add full distributed tracing to a 3-service system.