Pipes

Source: Dev.to

The problem

I wanted to run a pipeline in Go that behaves like the shell command:

tail -f test.log | head -3

tail -f never exits on its own, so the pipeline must handle the situation where the first command keeps running while the second (head) finishes after reading the required number of lines.

Demonstration in the shell

# In one terminal
echo line1 > test.log
tail -f ./test.log | head -3

In another terminal:

echo line2 >> ./test.log

head prints the line as soon as it appears. When a third line is written, head exits after receiving its third line. On the fourth write, tail receives a SIGPIPE because the read end of the pipe has been closed, causing tail to terminate.

The Bash documentation confirms this behavior:

The output of each command in the pipeline is connected via a pipe to the input of the next command. Each command reads the previous command’s output.
Each command in a multi‑command pipeline is executed in its own subshell, which is a separate process.
— Bash Reference Manual – Pipelines

Mimicking the behavior in Go

Creating the commands

cmd1 := exec.Command("tail", "-f", "./test.log")
cmd2 := exec.Command("head", "-n", "5")

Connecting the pipeline

outPipe, err := cmd1.StdoutPipe()
if err != nil {
    // handle error
}
cmd2.Stdin = outPipe
cmd2.Stdout = os.Stdout // forward head's output to the Go program's stdout

Starting the processes

if err := cmd1.Start(); err != nil {
    // handle error
}
if err := cmd2.Start(); err != nil {
    // handle error
}

Waiting for completion

cmd2 (the head command) will finish first. After it exits we must close the read end of the pipe in the parent process; otherwise the pipe remains open and tail never receives SIGPIPE.

cmd2.Wait()          // wait for head to finish
outPipe.Close()      // close the pipe in the parent
cmd1.Wait()          // now tail can exit

Complete example

package main

import (
    "fmt"
    "os"
    "os/exec"
)

func main() {
    cmd1 := exec.Command("tail", "-f", "/Users/skuf_love/study/go/test_pipes/test.log")
    cmd2 := exec.Command("head", "-n", "5")

    outPipe, err := cmd1.StdoutPipe()
    if err != nil {
        fmt.Printf("StdoutPipe error: %v\n", err)
        return
    }

    cmd2.Stdin = outPipe
    cmd2.Stdout = os.Stdout

    if err = cmd1.Start(); err != nil {
        fmt.Printf("cmd1 start error: %v\n", err)
        return
    }
    if err = cmd2.Start(); err != nil {
        fmt.Printf("cmd2 start error: %v\n", err)
        return
    }

    // Wait for head to finish, then close the pipe so tail receives SIGPIPE
    cmd2.Wait()
    fmt.Println("cmd2 wait done")
    outPipe.Close()
    cmd1.Wait()
    fmt.Println("cmd1 wait done")
}

This program reproduces the shell pipeline behavior: head reads the first five lines written to test.log, exits, and tail receives a SIGPIPE and terminates.