OpenTelemetry Filelog Receiver: A Guide to Ingesting Log Files

Source: Dev.to

How the Filelog receiver works

Before diving into configuration details, picture how the receiver handles a log file throughout its lifecycle. It operates as a simple repeating four‑step loop:

• Discover: Scans the filesystem at regular intervals, using the include and exclude patterns you set, to determine which log files to monitor.
• Read: Opens each selected file and follows it as new lines are written. The start_at setting decides whether to begin from the beginning or to tail new content from the end.
• Parse: Each line (or block of lines, if multiline parsing is used) runs through a series of Stanza operators (if configured). Operators parse the raw text, extract key attributes, assign timestamps and severity levels, and structure the log data.
• Emit: The structured log records are passed into the Collector’s pipeline, where they can be filtered, transformed further, or exported to your backend.

The Discover → Read → Parse → Emit loop forms the foundation of everything the receiver does.

Quick Start: tailing a log file

One common use case is an application that already writes logs in JSON format to a file, e.g. /var/log/myapp/app.log:

{"time":"2025-09-28 20:15:12","level":"INFO","message":"User logged in successfully","user_id":"u-123","source_ip":"192.168.1.100"}
{"time":"2025-09-28 20:15:45","level":"WARN","message":"Password nearing expiration","user_id":"u-123"}

Minimal filelog receiver configuration

# otelcol.yaml
receivers:
  filelog:
    # 1. DISCOVER all .log files in /var/log/myapp/
    include: [/var/log/myapp/*.log]
    # 2. READ from the beginning of new files
    start_at: beginning
    # 3. PARSE using the json_parser operator
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: "%Y-%m-%d %H:%M:%S"
        severity:
          parse_from: attributes.level

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [debug]

Configuration breakdown

• include: Points the receiver to all .log files in /var/log/myapp/.
• start_at: beginning: Processes the entire file the first time it’s seen. The default (end) would capture only new lines after the Collector starts.
• operators: Here we use a single json_parser to interpret each line as JSON and promote selected fields into the log record’s core metadata.
• timestamp and severity: Extract the time and level fields from the JSON and map them to OpenTelemetry’s top‑level Timestamp and Severity* fields.

With the debug exporter, you’ll see the parsed and structured output:

LogRecord #0
ObservedTimestamp: 2025-09-28 20:48:36.728437503 +0000 UTC
Timestamp: 2025-09-28 20:15:12 +0000 UTC
SeverityText: INFO
SeverityNumber: Info(9)
Body: Str({"time":"2025-09-28 20:15:12","level":"INFO","message":"User logged in successfully","user_id":"u-123","source_ip":"192.168.1.100"})
Attributes:
     -> user_id: Str(u-123)
     -> source_ip: Str(192.168.1.100)
     -> log.file.name: Str(myapp.log)
     -> time: Str(2025-09-28 20:15:12)
     -> level: Str(INFO)
     -> message: Str(User logged in successfully)
Trace ID:
Span ID:
Flags: 0

The raw JSON logs have now been converted into OpenTelemetry’s unified log data format, providing a consistent foundation for cross‑system observability.

Adding the full file path

The receiver automatically adds a log.file.name attribute. To also capture the full path, enable include_file_path:

# otelcol.yaml (excerpt)
receivers:
  filelog:
    include: [/var/log/myapp/*.log]
    include_file_path: true

Resulting attributes:

Attributes:
     -> log.file.path: Str(/var/log/myapp/app.log)
     -> log.file.name: Str(app.log)

More enrichment options are described in the official OpenTelemetry Filelog receiver documentation.

Filtering and managing log files

The fundamental step in configuring the filelog receiver is specifying which files to monitor via include and exclude glob patterns. The receiver first generates a list using include, then removes any matches from that list using exclude.

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/apps/**/*.log]
    exclude:
      - /var/log/apps/**/debug.log
      - /var/log/apps/**/*.tmp

In this example, every .log file under /var/log/apps/ (including subdirectories) is collected, except files named debug.log and any file ending with .tmp.

Excluding files by modification age

If the log directory you’re reading contains many existing log files, you can instruct the receiver to ignore files th