Day 18: FluentBit

Phase 6 · Sep 19, 2026

Agenda (2–3 hours)

• Read (45 min): FluentBit architecture documentation; INPUT, FILTER, OUTPUT plugins; FluentBit Kubernetes DaemonSet
• Study (45 min): FluentBit vs Fluentd — when do you need Fluentd's plugin ecosystem vs FluentBit's low-resource footprint?
• Practice (45 min): Run FluentBit as an ECS sidecar; collect stdout from the main container; parse JSON logs; forward to CloudWatch Logs and OpenSearch simultaneously
• Challenge (30 min): A FluentBit DaemonSet on EKS is consuming 500MB RAM per node. Profile which INPUT plugins are the heaviest; design a configuration to reduce memory to under 100MB

FluentBit Pipeline

INPUT → PARSER → FILTER → OUTPUT

[INPUT]
    Name   tail
    Path   /var/log/containers/*.log
    Tag    kube.*

[FILTER]
    Name   parser
    Match  kube.*
    Key_Name log
    Parser json

[FILTER]
    Name   record_modifier
    Match  kube.*
    Record cluster_name my-cluster

[OUTPUT]
    Name   cloudwatch_logs
    Match  kube.*
    region us-east-1
    log_group_name /ecs/task-svc
    log_stream_prefix ecs/

[OUTPUT]
    Name   opensearch
    Match  kube.*
    Host   my-opensearch.us-east-1.es.amazonaws.com
    Port   443
    tls    On
    Index  task-svc

JSON Parser

[PARSER]
    Name   json
    Format json
    Time_Key   timestamp
    Time_Format %Y-%m-%dT%H:%M:%SZ
    Time_Keep   On

After parsing, each log line's JSON fields become first-class record fields.
FluentBit can then filter on level = ERROR or add fields from the pod metadata.

Memory Management

[SERVICE]
    Flush        5
    Grace        30
    Mem_Buf_Limit 50MB   # back-pressure: stop tailing when buffer full

[INPUT]
    Name       tail
    Mem_Buf_Limit 10MB
    storage.type filesystem  # spill to disk instead of blocking

storage.type filesystem: when memory limit is hit, FluentBit writes to local disk rather than dropping logs or blocking the application.

Key Takeaways

• FluentBit's INPUT → FILTER → OUTPUT pipeline decouples log collection from routing
• JSON parser promotes log fields to record attributes for filtering and enrichment
• Mem_Buf_Limit + storage.type filesystem prevents OOM-kill during log bursts
• One FluentBit DaemonSet per EKS node; one sidecar per ECS task family

Tomorrow: log-based alerting and correlating traces, metrics, and logs.