Linux Kernel Deep Dive: Zombie Processes and Modern Process Management

Source: Dev.to

When you type ps aux, have you ever noticed the eerie letter “Z” in the STAT column?
Or have you ever encountered a “PID limit reached” error while investigating a CrashLoopBackOff on a Kubernetes pod?

These are all the work of zombie processes.

They consume neither memory nor CPU, but they cling to the precious resource known as a process ID (PID), wandering the system unable to rest in peace.

In this article we will unravel why processes turn into zombies after death, and how modern mechanisms like sub‑reaper and Go’s os/exec package reap them at the system‑call level.

1. The Identity of Zombie Processes – Why Do “Corpses” Remain?

In Linux, even when a process calls exit() and terminates, it does not disappear immediately.
The kernel retains a minimal amount of information (PID, exit status, CPU‑time usage, etc.) in the process table until the parent asks, “For what reason did that child die?”

This state – “dead but awaiting reporting to the parent” – is the zombie process.

Normal Life‑Cycle

If the parent calls wait(), the zombie is reaped and disappears.

Syscall cheat‑sheet

2. Scenarios Where Zombies Multiply – Child Neglect

The problem appears when the parent does not call wait().

• If the parent ignores the SIGCHLD signal (e.g., missing a signal handler), the child remains a zombie forever.

PID Exhaustion by cgroup

Historically the system’s pid_max (default 32 768) was the upper limit.
In the container era (Kubernetes) PID limits are enforced at the cgroup level (pids.max).

When the limit defined in a pod’s cgroup is reached, the container can no longer fork() (returns EAGAIN – Resource temporarily unavailable), even if the host still has free PIDs. A massive outbreak of zombies can quickly exhaust this quota.

Note – cgroups are a kernel feature that isolates and limits resources (CPU, memory, PIDs, etc.) for a group of processes. This is the reality behind “resource limits” in Docker and Kubernetes.

3. Orphan Processes and the Mercy of init (and Sub‑reaper)

What happens if the parent that spawned the zombie dies itself?

Collection by the init Process (PID 1)

When a parent dies, its children become orphan processes and are automatically adopted by the init process (PID 1).
Modern init systems (e.g., systemd) call wait() on any adopted child, allowing the zombie to be reaped.

Child Sub‑reaper

Since Linux 3.4, a process can declare itself a sub‑reaper:

prctl(PR_SET_CHILD_SUBREAPER, 1);

The calling process says, “I will adopt any orphaned descendants within my hierarchy.”

Docker, containerd, and Chrome’s process manager use this feature to collect orphaned processes without overloading PID 1.

4. Implementation Patterns to Avoid / Clear Zombies

Pattern A – “Honestly Wait” (Basic)

In the SIGCHLD handler, loop over waitpid(-1, &status, WNOHANG) to reap all terminated children.

void sigchld_handler(int sig) {
    int status;
    while (waitpid(-1, &status, WNOHANG) > 0) {
        /* child reaped */
    }
}

Pattern B – Double Fork (Daemonisation)

When to use: The parent deliberately wants to detach from its child (e.g., creating a daemon).

1. Parent forks a child and immediately wait()s for it.
2. Child forks a grandchild (the actual worker) and then exit().
3. Grandchild becomes an orphan, adopted by init (or a sub‑reaper), and will be reaped automatically.

Caution: In a service managed by systemd (Type=simple), double‑forking can cause systemd to lose track of the main process. Use manual daemonisation only when necessary.

Pattern C – tini (or dumb‑init) in Containers

When an application runs as PID 1 inside a Docker container, default signal handling and wait() may not work correctly.
Using a tiny init system such as tini or dumb‑init as the container entrypoint solves the “PID 1 problem”.

FROM alpine
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["my‑app"]

5. Language Deep‑Dive: Go’s os/exec – Concealed Reaping

The Go standard library’s os/exec package makes it easy to run external commands, but the underlying wait system call is hidden from the user.

What cmd.Run() Does

// Simplified internal logic of Cmd.Run()
func (c *exec.Cmd) Run() error {
    if err := c.Start(); err != nil {   // fork + exec
        return err
    }
    return c.Wait()                    // blocks, then reaps the child
}

• Start() performs a fork/exec (via the clone syscall).
• Wait() blocks until the child exits and then calls the appropriate waitpid to reap it.

If you call Start() without a subsequent Wait(), the child will become a zombie once it exits. Therefore, always pair Start() with Wait() (or use Run(), which does both).

TL;DR

• Zombie = dead process that still occupies a PID because the parent hasn’t called wait().
• Orphan = parent died; the child is adopted by init (PID 1) or a sub‑reaper.
• Avoid zombies by handling SIGCHLD and always reaping children (waitpid/wait).
• In containers, use a proper init (tini, dumb‑init) or rely on Docker’s sub‑reaper.
• In Go, never forget to call Cmd.Wait() after Cmd.Start().

Proper Use of cmd.Start() and cmd.Wait() in Go

if err := c.Start(); err != nil { // Internally clone + execve
    return err
}
return c.Wait() // Calls wait4 system call here!

The Trap: Using only cmd.Start()

If you call only cmd.Start() for asynchronous processing and forget cmd.Wait(), the child process will become a zombie and remain as long as the Go program is running.

// 💀 Zombie generation code
func spawnAndForget() {
    cmd := exec.Command("sleep", "1")
    cmd.Start() // Not calling Wait()!
    // 1 second later, the sleep command becomes a zombie (Z),
    // and remains until this Go program terminates.
}

Go’s garbage collector may collect the cmd struct in memory, but it will not collect the process status (zombie) on the OS.

If you use Start(), you must explicitly call Wait(), for example in a separate goroutine.

// ✅ Correct async execution
func spawnAsync() {
    cmd := exec.Command("sleep", "1")
    if err := cmd.Start(); err != nil {
        return
    }

    // Wait for collection in a goroutine
    go func() {
        cmd.Wait() // wait4 is called here, and the zombie rests in peace
    }()
}

What is happening at the low layer?

In a Linux environment, cmd.Wait() ultimately calls syscall.Wait4. This receives the exit status from the kernel and stores it in the ProcessState struct.

While the Go runtime handles signals skillfully, for individual external command execution the design requires developers to explicitly issue the wait system call via the Wait method.

Conclusion

The existence of zombie processes is a specification, not a bug, but leaving them unattended is a “child neglect” bug.

In 2026 troubleshooting, it is required not only to use the ps command but also to monitor who is skipping wait or which container is about to reach the PID limit using bpftrace. Trace the process tree (pstree -p), identify the parent process failing to fulfill its responsibilities, and fix it.