Day 17: OpenSearch and Log Indexing

Phase 6 · Sep 18, 2026

Agenda (2–3 hours)

• Read (45 min): Amazon OpenSearch Service documentation; index templates and mappings; OpenSearch Dashboards (formerly Kibana)
• Study (45 min): When does CloudWatch Logs Insights become insufficient and OpenSearch becomes necessary? What are the cost and operational trade-offs?
• Practice (45 min): Configure a FluentBit sidecar to ship ECS task logs to OpenSearch; create an index pattern; build a Discover search for error events
• Challenge (30 min): OpenSearch indexes every field by default, causing "mapping explosion" for high-cardinality JSON logs. Design a mapping that indexes only the fields you query

CloudWatch Logs vs OpenSearch

Use CloudWatch for operational logs; OpenSearch for long-term analysis and full-text search.

Index Mapping (Avoid Explosion)

{
  "index_patterns": ["task-svc-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "timestamp":   { "type": "date" },
        "level":       { "type": "keyword" },
        "message":     { "type": "text" },
        "trace_id":    { "type": "keyword" },
        "user_id":     { "type": "keyword" },
        "duration_ms": { "type": "float" },
        "http_status": { "type": "integer" }
      }
    }
  }
}

"dynamic": "strict" — unknown fields are rejected, not auto-indexed.

Index Lifecycle Policy

{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_age": "1d", "max_size": "50gb" } } },
      "warm":   { "min_age": "7d",  "actions": { "shrink": { "num_shards": 1 } } },
      "cold":   { "min_age": "30d", "actions": { "freeze": {} } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

Hot → warm → cold → delete transitions reduce storage cost while retaining searchability.

Key Takeaways

• OpenSearch excels at full-text search and aggregations over long retention windows
• Strict mapping prevents cardinality explosion from dynamic fields
• Index lifecycle policies automatically migrate and expire indices to control storage cost
• CloudWatch → OpenSearch via subscription filter or FluentBit; don't run both simultaneously without a filter

Tomorrow: FluentBit — collecting, filtering, and routing logs from containers.