Day 19: Istio Traffic Management

Phase 3 · Jul 19, 2026

Agenda (2–3 hours)

• Read (45 min): Istio traffic management documentation (retries, timeouts, circuit breaking, fault injection)
• Study (45 min): Design a canary deployment using VirtualService weight-based routing; think through how to do a safe rollback
• Practice (45 min): Apply a VirtualService with: 5-second timeout, 3 retries on 5xx, 10% fault injection; verify behavior with curl loops
• Challenge (30 min): Design a traffic management strategy for a zero-downtime API version migration

Retries and Timeouts in VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: reviews }
spec:
  hosts: [reviews]
  http:
  - timeout: 5s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: "5xx,reset,connect-failure,retriable-4xx"
    route:
    - destination: { host: reviews, subset: v1 }

retryOn: comma-separated list of retry conditions. retriable-4xx retries on 429 (rate limited).

Circuit Breaking in DestinationRule

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata: { name: reviews }
spec:
  host: reviews
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
    connectionPool:
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000

Circuit breaking at the mesh level: no code changes in services.

Fault Injection for Chaos Testing

http:
- fault:
    delay:
      percentage: { value: 10 }  # 10% of requests
      fixedDelay: 7s             # get a 7-second delay
    abort:
      percentage: { value: 5 }   # 5% of requests
      httpStatus: 503            # return 503 immediately
  route:
  - destination: { host: reviews }

Fault injection is the controlled version of chaos engineering: inject known failures to verify that your retries, timeouts, and circuit breakers work correctly before they're needed in production.

Canary Deployments

# 90% to stable v1, 10% to canary v2
http:
- route:
  - destination: { host: myapp, subset: v1 }
    weight: 90
  - destination: { host: myapp, subset: v2 }
    weight: 10

Increase v2 weight incrementally: 10 → 25 → 50 → 75 → 100 as metrics validate correctness. Rollback: set v2 weight to 0. No redeployment required — just update the VirtualService.

Key Takeaways

• VirtualService handles retries, timeouts, fault injection — configured once, applied to all callers
• DestinationRule handles circuit breaking and connection pool limits
• Fault injection enables chaos testing without modifying service code
• Canary deployments via weighted routing: gradual traffic shift with instant rollback

Tomorrow: observability in service mesh — tracing, metrics, and access logs from Envoy.