Pipes

What a pipe is

A pipe is the oldest, simplest form of inter-process communication (IPC) — the umbrella term for the OS mechanisms that let separate processes exchange data (pipes, sockets, shared memory, signals). A pipe specifically is a one-directional, in-kernel byte stream with a write end and a read end. Bytes written to one end come out the other, in order, with no message boundaries. There are two flavors:

• Anonymous pipe — no name; shared by handing a file descriptor to a child process. This is what the shell’s | builds.
• Named pipe (FIFO) — has a filesystem path, so unrelated processes can connect by opening it.

Go also gives a pure-Go, in-process pipe (io.Pipe) for streaming between goroutines.

In-process streaming: io.Pipe

io.Pipe() connects an io.Reader and io.Writer in memory — perfect for feeding one API’s output straight into another’s input without a temp buffer. It’s synchronous (a write blocks until a read consumes it), so it runs on the playground:

package main

import (
"fmt"
"io"
"strings"
)

func main() {
pr, pw := io.Pipe() // reader end, writer end

// Producer goroutine writes, then closes the write end (→ EOF for reader).
go func() {
	defer pw.Close()
	for i := 1; i <= 3; i++ {
		fmt.Fprintf(pw, "line %d\n", i) // blocks until the reader reads
	}
}()

// Consumer reads the stream to completion.
var sb strings.Builder
io.Copy(&sb, pr) // reads until the write end is closed
fmt.Print(sb.String())
}

The classic use: stream a large response body through a transform (compress, hash, re-encode) into an uploader, with constant memory — the writer can’t outrun the reader because each Write waits for a Read.

Real kernel pipes: os.Pipe

os.Pipe() returns two real *os.Files backed by an actual kernel pipe — so you can hand the write end to a child process:

pr, pw, _ := os.Pipe()        // real fds

cmd := exec.Command("wc", "-l")
cmd.Stdin = pr                // child reads from the pipe
go func() {
	defer pw.Close()
	fmt.Fprintln(pw, "one")   // we write...
	fmt.Fprintln(pw, "two")
}()
out, _ := cmd.Output()        // ...child counts the lines → "2"

Often you don’t even need os.Pipe directly: exec wires pipes for you when you connect commands.

You can also use the two ends in one process — this is a real kernel pipe (the pipe(2) syscall), with actual file descriptors, and it runs here:

package main

import (
"bufio"
"fmt"
"os"
)

func main() {
pr, pw, _ := os.Pipe() // two *os.File over a real kernel pipe
fmt.Printf("read fd = %d, write fd = %d\n", pr.Fd(), pw.Fd())

// Producer goroutine writes to the write end, then closes it (→ EOF).
go func() {
	defer pw.Close()
	for i := 1; i <= 3; i++ {
		fmt.Fprintf(pw, "line %d\n", i)
	}
}()

// Consumer reads the read end until EOF.
sc := bufio.NewScanner(pr)
for sc.Scan() {
	fmt.Println("got:", sc.Text())
}
}

Unlike io.Pipe (pure Go, in-process), these fds are real — you could pass the write end to a child process, and strace would show the pipe2/read/write calls.

graph LR
A["producer (write end)"] -->|"kernel pipe buffer"| B["consumer (read end)"]
B -. "buffer full → writer blocks (back-pressure)" .-> A

Connecting two processes

To reproduce ls | wc -l in Go, connect one command’s StdoutPipe to another’s Stdin:

ls := exec.Command("ls", "-1")
wc := exec.Command("wc", "-l")

wc.Stdin, _ = ls.StdoutPipe() // ls's stdout → wc's stdin
wc.Stdout = os.Stdout

wc.Start()
ls.Run()      // run the producer
wc.Wait()     // wait for the consumer

Named pipes (FIFOs)

When the processes aren’t parent/child, give the pipe a name in the filesystem:

// One-time: create the FIFO (or `mkfifo /tmp/events` in the shell).
syscall.Mkfifo("/tmp/events", 0o644)

// Writer process:
w, _ := os.OpenFile("/tmp/events", os.O_WRONLY, 0)
fmt.Fprintln(w, "event: deploy")

// Reader process (any unrelated program):
r, _ := os.Open("/tmp/events")
io.Copy(os.Stdout, r)

A FIFO looks like a file (ls -l shows a p) but behaves like a pipe: open blocks until both a reader and writer are present, and it carries a stream, not stored contents.

See also

• Unix sockets — bidirectional local IPC when pipes aren’t enough.
• processes & exec — wiring pipes between commands.
• the pipeline pattern — the channel-based, in-process analog.
• fmt & io (stdlib) — the io.Reader/io.Writer interfaces pipes implement.

Next: bidirectional local IPC — Unix sockets.

Related topics