Why Your Python Logs Vanish in Docker (and How PYTHONUNBUFFERED=1 Saves the Day)

Source: Dev.to

Demo Application

I create a neat little Python app. It runs perfectly, printing helpful logs at every step:

# app.py
print("Starting application...")
print("Connecting to database...")
print("Processing request...")
print("All done!")

Running it locally:

$ python app.py
Starting application...
Connecting to database...
Processing request...
All done!

All good! I containerize it:

FROM python:3.12-slim
COPY app.py .
CMD ["python", "app.py"]

Build and run it in Docker:

$ docker build -t myapp .
$ docker run myapp
$ docker logs 
# (no output)

But the logs are nowhere to be seen. Where did they go? If running on Kubernetes, kubectl logs shows the same thing—no logs.

The logs, like a magician’s trick, have vanished into thin air. What sorcery is this?

Spoiler: It’s Python’s output buffering, and it’s been frustrating developers since Docker became popular.

What is Output Buffering?

Output buffering is when your program doesn’t immediately write output to its destination (terminal, Docker logs, etc.). Instead, it collects output in a temporary buffer (a chunk of memory) and writes it all at once.

Without buffering

[Your code] ---> "Hello" ---> [Terminal]
            ---> "World" ---> [Terminal]
            ---> "!!!"   ---> [Terminal]

With buffering

[Your code] ---> "Hello" --\
            ---> "World" ---}--[Buffer]---> [Terminal] (when full or flushed)
            ---> "!!!"   --/

Why Buffering Exists

Buffering is a performance optimization. Writing to output (terminal, files, network) is slow compared to memory operations.

The Three Types of Buffering

Python (and most languages) uses three buffering modes:

Python detects whether stdout is attached to a terminal. If it is, it line‑buffers; otherwise it block‑buffers.

Want to see the actual buffer size? You can check it yourself:

import io
print(f"Default buffer size: {io.DEFAULT_BUFFER_SIZE} bytes")
# Output: Default buffer size: 8192 bytes

io.DEFAULT_BUFFER_SIZE (defined in Python’s io module) is what Python uses for block‑buffering. It’s typically 8192 bytes (8 KB), though it can vary based on the system’s block size.

Why Docker Makes This Worse

When you run Python in a Docker container:

• stdout is not a TTY (it’s a pipe to Docker’s logging driver).
• Python therefore switches to block‑buffering (8 KB buffer by default).

The logs sit in memory until:

• The buffer fills up, or
• The program exits, or
• You explicitly flush.

It’s a classic “works on my machine” case—your local machine is a TTY, Docker isn’t.

The Solution: PYTHONUNBUFFERED=1

Set the PYTHONUNBUFFERED environment variable to any non‑empty value. This forces Python to use unbuffered mode for stdout and stderr.

In a Dockerfile

FROM python:3.12-slim

# Add this line!
ENV PYTHONUNBUFFERED=1

COPY app.py .
CMD ["python", "app.py"]

In docker‑compose.yml

version: '3.8'
services:
  app:
    build: .
    environment:
      - PYTHONUNBUFFERED=1

In a Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: myapp:latest
          env:
            - name: PYTHONUNBUFFERED
              value: "1"

At Runtime

docker run -e PYTHONUNBUFFERED=1 myapp

Now the logs appear instantly:

$ docker logs myapp
Starting application...
Connecting to database...
Processing request...
All done!

Fun fact: The value doesn’t matter. PYTHONUNBUFFERED=1, PYTHONUNBUFFERED=true, even PYTHONUNBUFFERED=banana all work. Python just checks that the variable is set and non‑empty.

Alternative Solutions

Use Python’s unbuffered command‑line option

python -u app.py

Flush manually

print("Processing request...", flush=True)

or

import sys
print("Processing request...")
sys.stdout.flush()

Use stderr in Python’s logging module

import logging
logging.basicConfig(stream=sys.stderr, level=logging.INFO)
logging.info("Processing request...")

Conclusion

When Python runs inside a container, its stdout is block‑buffered, causing logs to disappear until the buffer fills or the process ends. Setting PYTHONUNBUFFERED=1 (or using -u, manual flushing, or stderr) disables buffering and restores real‑time log visibility.

References

• Python documentation – IO buffering
• Docker documentation – Logging drivers
• Kubernetes documentation – Container environment variables

Alternative Solutions

While PYTHONUNBUFFERED=1 is the easiest fix, here are other approaches:

Use python unbuffered command‑line option

The -u flag forces unbuffered mode:

FROM python:3.12-slim
COPY app.py .
CMD ["python", "-u", "app.py"]  # Note the -u flag

This is equivalent to PYTHONUNBUFFERED=1, just more explicit.

Pros

• No environment variable needed

Cons

• Have to remember to add -u everywhere you run Python

---

Flush manually

Call flush() explicitly when you want output to appear:

import sys

print("Critical log message", flush=True)  # Appears immediately

# Or flush everything:
sys.stdout.flush()

Pros

• Fine‑grained control over when to flush

Cons

• Easy to forget
• Clutters your code
• Doesn’t help with third‑party libraries that use print()

---

Use stderr in Python’s logging module

stderr is unbuffered by default.

import logging
import sys

# Configure logging to stderr (which is unbuffered by default)
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    stream=sys.stderr  # stderr is unbuffered!
)

logger = logging.getLogger(__name__)

logger.info("Application starting")
logger.info("Connected to database")
logger.error("Something went wrong!")

Why this works

• stderr is unbuffered by default (even in Docker)
• Structured logging is better for production anyway
• You get timestamps, log levels, and better formatting

---

Conclusion

The PYTHONUNBUFFERED=1 environment variable is a tiny fix that solves a huge headache. It forces Python to stop buffering output, so your logs appear immediately in Docker and Kubernetes.

References

• Python io Module Documentation (DEFAULT_BUFFER_SIZE)
• Python Documentation: -u flag
• Python io Module: Text I/O Buffering
• Python Logging HOWTO
• Python sys.stdout Buffering
• The TTY Demystified

This article was originally published on my blog at why‑your‑python‑logs‑vanish‑in‑docker‑pythonunbuffered‑explained.

Find more of my content on my blog, where I share my software development experiences and learnings: wewake.dev