15. Pipes and File Redirection

Pipes and File Redirection#

As Processes explains, processes normally have separate address spaces. One process cannot simply dereference a pointer into another process's ordinary memory. Inter-process communication, or IPC, provides controlled ways for processes to exchange data.

A pipe is one of the simplest IPC mechanisms: a kernel-managed, unidirectional stream of bytes. One side writes bytes, and the other side reads them in the same order.

flowchart LR
    A["Producer process"] -->|"write end"| P[["pipe buffer"]]
    P -->|"read end"| B["Consumer process"]

This is the mechanism behind a shell pipeline such as:

seq 1 10 | wc

seq writes to its standard output. The shell arranges for that descriptor to refer to the pipe. wc reads its standard input, which the shell has arranged to refer to the other end.

Creating a Pipe#

#include <unistd.h>

int pipe(int pipefd[2]);

On success:

• pipefd[0] is the read end;
• pipefd[1] is the write end.

int pipefd[2];
if (pipe(pipefd) == -1) {
	perror("pipe");
	return 1;
}

The descriptors can be passed to ordinary read() and write(). A pipe contains bytes, not messages. If you want to transmit integers, records or lines, the processes must agree on a protocol.

A Parent Sending Data to a Child#

The usual pattern is to create the pipe before fork(). Both resulting processes inherit both descriptors; each must then close the end it does not use.

#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void) {
	int pipefd[2];
	if (pipe(pipefd) == -1) {
		perror("pipe");
		return 1;
	}

	pid_t pid = fork();
	if (pid == -1) {
		perror("fork");
		close(pipefd[0]);
		close(pipefd[1]);
		return 1;
	}

	if (pid == 0) {
		close(pipefd[1]);

		char buffer[128];
		ssize_t n = read(pipefd[0], buffer, sizeof buffer);
		if (n == -1) {
			perror("read");
			_exit(1);
		}
		printf("child received %zd bytes\n", n);
		fflush(stdout);

		close(pipefd[0]);
		_exit(0);
	}

	close(pipefd[0]);
	char message[] = "hello from the parent";
	if (write(pipefd[1], message, sizeof message) == -1) {
		perror("write");
	}
	close(pipefd[1]);

	waitpid(pid, NULL, 0);
	return 0;
}

The shape is:

sequenceDiagram
    participant P as Parent
    participant K as Pipe
    participant C as Child
    P->>P: close(read end)
    C->>C: close(write end)
    P->>K: write(bytes)
    P->>P: close(write end)
    K-->>C: read(bytes)
    K-->>C: EOF after buffer empties

Why Unused Ends Must Be Closed#

read() on an empty pipe does not immediately mean EOF. It blocks while any descriptor referring to the write end remains open, because another writer could still send bytes.

Suppose the child wants to read until EOF, but accidentally keeps its own inherited write end open. Even after the parent closes its writer, the kernel still sees one writer: the child's unused descriptor. The child waits forever for data which it could theoretically write to itself.

The rule is simple:

• every process closes every pipe end it will not use;
• writers close the write end after their final byte;
• readers treat read() == 0 as end-of-stream.

Closing is part of the communication protocol, not merely cleanup.

File Descriptor Duplication#

dup() and dup2() make another descriptor refer to the same open file description. This is the descriptor-to-open-file-description distinction introduced in File Systems:

int dup(int oldfd);
int dup2(int oldfd, int newfd);

dup() selects the lowest available descriptor. dup2(oldfd, newfd) uses the requested number, closing newfd first if necessary.

The duplicate descriptors share:

• the underlying file or pipe endpoint;
• the current file offset for a regular file;
• file status flags such as append mode.

flowchart TD
    subgraph S1["Stage 1: After open('input.txt', O_RDONLY)"]
        S1_0["fd 0: STDIN (keyboard)"] --> S1_K["Terminal Input stream"]
        S1_3["fd 3: input.txt"] --> S1_OFD["Open File Description: input.txt<br/>offset: 0 | read-only"]
        S1_OFD --> S1_F["input.txt on disk"]
    end

    subgraph S2["Stage 2: After dup2(3, 0)"]
        S2_0["fd 0: input.txt"] --> S2_OFD["Open File Description: input.txt<br/>offset: 0 | read-only"]
        S2_3["fd 3: input.txt"] --> S2_OFD
        S2_OFD --> S2_F["input.txt on disk"]
    end

    subgraph S3["Stage 3: After close(3) and execv(...)"]
        S3_0["fd 0: input.txt"] --> S3_OFD["Open File Description: input.txt<br/>offset advances as program reads"]
        S3_OFD --> S3_F["input.txt on disk"]
    end

A read through descriptor 0 advances the same shared offset in the underlying open file description. Because the open file description is preserved across execv(), the newly executed program reads seamlessly from input.txt whenever it reads from standard input (STDIN_FILENO).

Redirecting Standard Input and Output#

A program such as tr does not need special "read from a file" logic. It reads descriptor 0. Another process can replace descriptor 0 before executing it:

int fd = open("input.txt", O_RDONLY);
if (fd == -1) {
	perror("input.txt");
	return 1;
}

if (dup2(fd, STDIN_FILENO) == -1) {
	perror("dup2");
	return 1;
}
close(fd);

char *argv[] = {"/usr/bin/tr", "a-z", "A-Z", NULL};
execv(argv[0], argv);
perror("execv");

After dup2, both fd and 0 refer to the same open file description. We close the now-redundant original descriptor, then execv() replaces the program while descriptor 0 survives. The new program reads the file as though it were standard input.

Output redirection follows the same pattern:

int fd = open("message.txt", O_WRONLY | O_CREAT | O_TRUNC, 0666);
if (fd == -1) {
	perror("message.txt");
	return 1;
}

if (dup2(fd, STDOUT_FILENO) == -1) {
	perror("dup2");
	return 1;
}
close(fd);

char *argv[] = {"/bin/echo", "hello", "world", NULL};
execv(argv[0], argv);
perror("execv");

This is equivalent in spirit to echo hello world > message.txt.

Joining Two Programs with a Pipe#

To implement producer | consumer:

1. create a pipe;
2. create both child processes;
3. in the producer, duplicate the write end onto standard output;
4. in the consumer, duplicate the read end onto standard input;
5. close all original pipe descriptors in every process;
6. execute the respective programs;
7. let the parent wait for both.

flowchart LR
    PROD["producer<br/>stdout = pipe write end"] --> PIPE[["pipe"]]
    PIPE --> CONS["consumer<br/>stdin = pipe read end"]
    PARENT["parent"] -."waits for both".-> PROD
    PARENT -."waits for both".-> CONS

The producer and consumer should run concurrently. Waiting for the producer before starting the consumer can deadlock if the producer fills the finite pipe buffer and blocks while nobody is reading.

Using fdopen#

fdopen() associates a FILE * stream with an existing descriptor, bridging the low-level descriptor interface with the buffered streams from Files and Streams:

FILE *stream = fdopen(pipefd[0], "r");
if (stream == NULL) {
	perror("fdopen");
}

The mode must be compatible with the descriptor's access mode: for example, do not request "w" for a read-only pipe end. You may then use fgets() or fread(). Once the stream owns that descriptor, close it with fclose(stream). Do not separately close the same descriptor and later use the stream.

popen: Convenient but Risky#

FILE *popen(const char *command, const char *mode);
int pclose(FILE *stream);

With mode "r", your process reads the command's standard output. With "w", your process writes to its standard input.

FILE *stream = popen("/bin/date --utc", "r");
if (stream == NULL) {
	perror("popen");
	return 1;
}

char line[256];
if (fgets(line, sizeof line, stream) != NULL) {
	printf("captured: %s", line);
}

int status = pclose(stream);

Close a popen() stream with pclose(), not fclose(), because pclose() also waits for the command. It returns -1 on failure; otherwise its result is an encoded wait status which should be interpreted with macros such as WIFEXITED and WEXITSTATUS, just like the status produced by waitpid().

Like system(), popen() runs a shell command string. It is brittle and vulnerable to command injection if any untrusted text is inserted. Prefer explicit process creation and argument arrays for reliable programs.

Pipes with posix_spawn#

posix_spawn_file_actions_t describes descriptor changes which occur in the child before its new program begins:

#include <errno.h>
#include <spawn.h>

posix_spawn_file_actions_t actions;
int error = posix_spawn_file_actions_init(&actions);
if (error != 0) {
	errno = error;
	perror("posix_spawn_file_actions_init");
	return 1;
}

if ((error = posix_spawn_file_actions_adddup2(
		&actions, pipefd[1], STDOUT_FILENO)) != 0
	|| (error = posix_spawn_file_actions_addclose(
		&actions, pipefd[0])) != 0
	|| (error = posix_spawn_file_actions_addclose(
		&actions, pipefd[1])) != 0) {
	errno = error;
	perror("posix_spawn file action");
	posix_spawn_file_actions_destroy(&actions);
	return 1;
}

error = posix_spawn(&pid, path, &actions, NULL, argv, environ);
int destroy_error = posix_spawn_file_actions_destroy(&actions);

if (error != 0) {
	errno = error;
	perror("posix_spawn");
	return 1;
}
if (destroy_error != 0) {
	errno = destroy_error;
	perror("posix_spawn_file_actions_destroy");
	return 1;
}

The API is verbose, but it avoids constructing a shell command and expresses redirection directly. Always destroy the action object after the spawn attempt and close the parent's copies of descriptors it no longer needs.

Blocking and Broken Pipes#

Pipe behaviour follows the presence of readers and writers:

• reading from an empty pipe blocks while a writer remains;
• reading from an empty pipe returns 0 once all write ends are closed;
• writing may block when the pipe buffer is full;
• writing when no read end remains generates SIGPIPE by default, or fails with EPIPE if the signal is handled or ignored.

This is why descriptor lifetime controls program lifetime. A forgotten writer prevents EOF; a forgotten reader can keep a producer from discovering that its output is unwanted.

Common Mistakes#

• Reversing pipefd[0] and pipefd[1].
• Creating the pipe after fork(), leaving the processes with unrelated pipes.
• Forgetting to close unused ends in either parent or child.
• Waiting for the producer before starting the consumer.
• Forgetting that a pipe is a byte stream, not a packet or C-struct channel.
• Calling fclose() instead of pclose() on a popen() stream.
• Passing untrusted input through popen() or system().
• Forgetting that duplicated descriptors share an offset.

Practice#

1. Which index in int pipefd[2] is used for reading, and which for writing?
2. What happens if a child process attempts to read from a pipe until EOF, but forgot to close its own copy of the write end?
3. What does dup2(oldfd, newfd) do if newfd is already open?
4. What signal is delivered to a process that attempts to write to a pipe with no active read ends?
5. Why is pclose() required instead of fclose() when closing a stream created with popen()?