Architecting Observability in Kubernetes with OpenTelemetry and Fluent Bit

Source: DZone DevOps

Why OpenTelemetry (OTel) and Fluent Bit?

• OpenTelemetry is a vendor‑agnostic, CNCF‑hosted framework that standardizes the collection of metrics, logs, and traces. It provides SDKs, APIs, and agents that can be instrumented directly into applications or run as sidecars/daemonsets.
• Fluent Bit is a lightweight, high‑performance log processor and forwarder. It can ingest logs from containers, enrich them with Kubernetes metadata, and ship them to a variety of back‑ends (e.g., Elasticsearch, Loki, CloudWatch).

Combining OTel with Fluent Bit gives you a unified pipeline:

• OTel Collector gathers metrics and traces from the cluster, applies processors (e.g., batching, attribute enrichment), and exports them to observability platforms.
• Fluent Bit handles log collection, parsing, and routing, ensuring logs are correlated with the corresponding metrics and traces via shared identifiers (e.g., trace_id, span_id).

Architecture Overview

flowchart TD
    subgraph K8s Cluster
        A[Application Pods] -->|OTel SDK| B[OTel Collector (DaemonSet)]
        A -->|Fluent Bit Sidecar| C[Fluent Bit (DaemonSet)]
    end
    B -->|Metrics & Traces| D[Observability Backend]
    C -->|Logs| D
    D -->|Correlation UI| E[Dashboard / APM]

1. Instrumentation – Add OTel SDKs to your services (available for Go, Java, Python, .NET, etc.).
2. Collector Deployment – Deploy the OTel Collector as a DaemonSet with a configuration that:
   • Receives metrics via Prometheus scrapes.
   • Receives traces via OTLP over gRPC/HTTP.
   • Exports data to your chosen backend (e.g., Jaeger, Prometheus, Grafana Cloud).
3. Fluent Bit Deployment – Deploy Fluent Bit as a DaemonSet that:
   • Reads container logs from /var/log/containers/*.log.
   • Enriches logs with pod, namespace, and node metadata.
   • Adds OTel trace context (if present) to enable log‑trace correlation.
   • Sends logs to the same backend or a dedicated log store.

Step‑by‑Step Implementation

1. Add OpenTelemetry SDKs

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

func main() {
    // Initialize tracer provider
    tp := otel.GetTracerProvider()
    tracer := tp.Tracer("my-service")
    // Use tracer in your handlers...
}

Replace the language‑specific snippet with the appropriate SDK for your stack.

2. Deploy the OTel Collector

Create otel-collector-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'kubernetes-pods'
              kubernetes_sd_configs:
                - role: pod
    processors:
      batch:
      memory_limiter:
        limit_mib: 400
        spike_limit_mib: 100
        check_interval: 5s
    exporters:
      otlphttp:
        endpoint: "https://api.myobservability.com/v1/traces"
      prometheusremotewrite:
        endpoint: "https://api.myobservability.com/v1/metrics"
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp]
        metrics:
          receivers: [prometheus]
          processors: [batch, memory_limiter]
          exporters: [prometheusremotewrite]

Apply the DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector:latest
          args: ["--config=/conf/otel-collector-config.yaml"]
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: otel-collector-config

3. Deploy Fluent Bit

Create fluent-bit-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: observability
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*

    [FILTER]
        Name              kubernetes
        Match             kube.*
        Merge_Log         On
        Keep_Log          Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name              record_modifier
        Match             *
        Record            trace_id ${TRACE_ID}
        Record            span_id ${SPAN_ID}

    [OUTPUT]
        Name  es
        Match *
        Host  elasticsearch.logging.svc
        Port  9200
        Index kubernetes-logs
        Type  _doc

Apply the DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: observability
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:latest
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config

Correlating Logs, Metrics, and Traces

• Trace Context Propagation – Ensure your services propagate traceparent and tracestate headers (W3C Trace Context). Fluent Bit extracts these headers from logs (when present) and adds them as trace_id and span_id fields.
• Unified Dashboards – In Grafana or your APM UI, you can click a trace ID to view related logs and metric graphs, providing end‑to‑end visibility.

Benefits of This Approach

Final Thoughts

Architecting observability in Kubernetes with OpenTelemetry and Fluent Bit gives you a single, coherent pipeline for all telemetry data. By standardizing collection, enrichment, and export, you reduce operational complexity, accelerate incident response, and lay a solid foundation for future scaling and compliance needs.

Start by instrumenting a few critical services, deploy the collector and Fluent Bit, and iteratively expand coverage. The payoff—clear, actionable insight into your cloud‑native applications—is well worth the effort.