12. File Systems

File Systems#

On a Unix-like system, a regular file is fundamentally a sequence of zero or more bytes. The filesystem gives that sequence a name, location and metadata, whilst the operating system provides controlled ways to open it and transfer bytes.

The file itself does not normally carry an authoritative declaration that it is “a JPEG”, “a C program” or “plain text”. A filename extension is a convention for humans and software. The meaning comes from the format in which a program interprets the bytes, just as the meaning of an integer bit pattern depends on its type in Integers.

From a Pathname to Bytes#

A persistent storage device contains blocks, not an inherent tree of familiar filenames. A filesystem maintains the structures which map pathnames to file data and metadata.

flowchart LR
    P["pathname<br/>notes/week5.md"] --> FS["filesystem lookup"]
    FS --> M["metadata<br/>type, size, permissions"]
    FS --> B["storage locations<br/>containing file bytes"]
    B --> RAM["bytes copied into RAM<br/>when read"]

The precise on-disk structures vary between filesystems, but programs do not ordinarily need to know them. They ask the kernel to resolve a pathname, and the kernel communicates with the filesystem and device driver.

A Unix file is a byte stream#

For a regular file, the OS exposes an ordered sequence of bytes and a size. Structure beyond that is imposed by the program:

• a text editor interprets some byte sequences as encoded characters;
• an image viewer interprets headers, dimensions and compressed pixel data;
• the loader interprets an executable format as code and data segments;
• a compiler interprets bytes as source-language tokens.

This is why opening a binary file in a text editor produces nonsense: the text editor is applying the wrong interpretation, not discovering that the file has “become corrupted”.

File Descriptors#

Before using read or write, a process normally obtains a file descriptor. A file descriptor is a small, non-negative integer used by that process to refer to an open file or stream.

The descriptor is best understood as an index into a per-process table maintained by the kernel. Its table entry refers to an open file description, which stores state including the current file offset and status flags.

flowchart TD
    subgraph P1["Process A Descriptor Table"]
        P1_0["0 (stdin)"]
        P1_1["1 (stdout)"]
        P1_2["2 (stderr)"]
        P1_3["3 (log.txt)"]
        P1_4["4 (data.bin)"]
    end

    subgraph OFD["Kernel Open File Description Table"]
        O_IN["stdin stream"]
        O_OUT["stdout stream"]
        O_ERR["stderr stream"]
        O_LOG["log.txt description<br/>offset: 128 | write-only | refcount: 1"]
        O_DATA1["data.bin description<br/>offset: 0 | read-only | refcount: 1"]
        O_DATA2["data.bin description<br/>offset: 512 | read-only | refcount: 1"]
    end

    subgraph P2["Process B Descriptor Table"]
        P2_0["0 (stdin)"]
        P2_1["1 (stdout)"]
        P2_2["2 (stderr)"]
        P2_3["3 (data.bin)"]
    end

    subgraph VNODE["Vnode / Inode Table (Files on Disk)"]
        V_LOG["inode 1042: log.txt"]
        V_DATA["inode 3891: data.bin"]
    end

    P1_0 --> O_IN
    P1_1 --> O_OUT
    P1_2 --> O_ERR
    P1_3 --> O_LOG
    P1_4 --> O_DATA1

    P2_0 --> O_IN
    P2_1 --> O_OUT
    P2_2 --> O_ERR
    P2_3 --> O_DATA2

    O_LOG --> V_LOG
    O_DATA1 --> V_DATA
    O_DATA2 --> V_DATA

Descriptors are local to a process. File descriptor 3 in Process A refers to log.txt, whereas descriptor 3 in Process B refers to data.bin. Furthermore, Process A and Process B can open the same file independently, producing two separate open file descriptions with their own independent file offsets.

The Linux open documentation describes the returned descriptor as the lowest currently unused non-negative integer and distinguishes the per-process descriptor from its system-wide open file description: open(2).

The current file offset#

For a seekable file, the open file description records a byte offset. A successful read or write normally begins there and advances it by the number of bytes transferred.

file bytes:   H e l l o ,   w o r l d !
offset:       0 1 2 3 4 5 6 7 8 9 ...
                           ^
                    current offset 6

After reading five bytes from offset 0, the next read begins at offset 5 unless the program changes the offset with lseek or another operation.

Standard Streams#

Every ordinary Unix process begins with three conventional descriptors:

They are treated through the same descriptor interface as opened files, pipes and many devices. A program which reads descriptor 0 does not need to know whether the bytes come from a keyboard, a file or another process.

Why standard error exists#

Separating ordinary output from diagnostics allows a user to redirect them independently:

./program >output.txt
./program 2>errors.txt
./program >output.txt 2>errors.txt

This matters for programs whose standard output is intended for another program. An error message mixed into a data stream could corrupt it.

In C, the corresponding <stdio.h> streams are stdin, stdout and stderr:

fprintf(stdout, "result: %d\n", result);
fprintf(stderr, "invalid input\n");

The File I/O Lifecycle#

Low-level file I/O generally follows four stages:

flowchart LR
    O["open(path, flags)"] --> FD["file descriptor"]
    FD --> IO["read / write<br/>zero or more times"]
    IO --> C["close(fd)"]

Each call can fail, so a correct program checks every return value rather than assuming that a valid-looking request succeeded.

Opening a File#

The open wrapper is declared in <fcntl.h>:

#include <fcntl.h>

int open(const char *path, int flags, ...);

On success it returns a file descriptor. On failure it returns -1 and sets errno.

Exactly one access mode is required:

Additional flags change how the open occurs:

Flags are combined with bitwise OR, as described in Bitwise Operations:

int fd = open("output.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);

The third mode argument is required when O_CREAT may create a file. It supplies permission bits, which are modified by the process's umask.

Permissions and octal#

Unix permissions are naturally written in octal because each octal digit represents the three read, write and execute bits for one permission class. Thus 0644 grants read and write to the owner and read-only access to the group and others. The leading 0 is C's octal prefix; decimal 644 would set an entirely different bit pattern. Octal Permissions develops the full mapping.

Opening safely#

#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>

int fd = open("input.txt", O_RDONLY);
if (fd == -1) {
    perror("input.txt");
    exit(EXIT_FAILURE);
}

perror uses the current errno value to append a human-readable reason.

Reading Bytes#

The read wrapper is declared in <unistd.h>:

#include <unistd.h>

ssize_t read(int fd, void *buffer, size_t count);

It attempts to place up to count bytes into the memory beginning at buffer.

Its return value has three distinct meanings:

The word “up to” is essential. A successful read may return fewer bytes than requested. This is common for pipes, terminals and network connections, and can also occur when a regular file ends. The Linux read(2) documentation specifies that the file offset advances by the number of bytes actually read for seekable files.

char buffer[4096];
ssize_t bytes_read = read(fd, buffer, sizeof buffer);

if (bytes_read == -1) {
    perror("read");
} else if (bytes_read == 0) {
    // end-of-file
} else {
    // exactly bytes_read bytes in buffer are valid
}

read does not append a null terminator. The result is a sequence of bytes, not automatically a C string. If text is intended, reserve an extra byte and terminate it yourself:

char buffer[4097];
ssize_t n = read(fd, buffer, 4096);
if (n > 0) {
    buffer[n] = '\0';
}

Writing Bytes#

The write wrapper is also declared in <unistd.h>:

ssize_t write(int fd, const void *buffer, size_t count);

It attempts to transfer count bytes beginning at buffer to the stream represented by fd.

A successful write may write fewer bytes than requested. Code which must emit the complete buffer needs a loop:

#include <errno.h>
#include <unistd.h>

ssize_t write_all(int fd, const void *buffer, size_t count) {
    const unsigned char *bytes = buffer;
    size_t written = 0;

    while (written < count) {
        ssize_t result = write(fd, bytes + written, count - written);

        if (result == -1) {
            if (errno == EINTR) {
                continue;
            }
            return -1;
        }

        // No progress with bytes still pending would otherwise loop forever.
        if (result == 0) {
            errno = EIO;
            return -1;
        }

        written += (size_t)result;
    }

    return (ssize_t)written;
}

This loop advances the pointer by the number of bytes already written and requests only the remaining count.

Seeking with lseek#

Sequential I/O moves the current offset forward by the number of bytes read or written. Seeking explicitly changes this file offset without transferring data.

The lseek system call is declared in <unistd.h>:

#include <unistd.h>

off_t lseek(int fd, off_t offset, int whence);

The whence parameter specifies how offset is interpreted:

On success, lseek returns the resulting file offset (in bytes from the beginning of the file). On failure, it returns (off_t)-1 and sets errno.

// Determine the total size of a file
off_t file_size = lseek(fd, 0, SEEK_END);
if (file_size == (off_t)-1) {
    perror("lseek");
}

// Rewind back to the beginning
if (lseek(fd, 0, SEEK_SET) == (off_t)-1) {
    perror("lseek");
}

Seeking past the end and sparse files#

POSIX allows seeking past the current end of a regular file. A read at that position returns 0 because the offset is beyond EOF. If data is later written there, the intervening gap reads back as zero bytes. Filesystems which represent that gap without allocating physical data blocks create a sparse file, or a file with a hole; POSIX specifies the observable zero bytes, not whether the storage is physically sparse.

Closing a Descriptor#

int close(int fd);

close releases the process's descriptor so the number can be reused. It returns 0 on success or -1 on error.

if (close(fd) == -1) {
    perror("close");
}

Closing is important because a process can have only a limited number of descriptors open. The kernel also needs to release its references and related state.

Do not continue using a descriptor after closing it. The same integer may soon be returned by another open and refer to a completely different object.

Error Reporting with errno#

Many libc system-call wrappers use the convention:

1. return a value such as -1 to indicate failure;
2. set errno to an error code describing the reason.

#include <errno.h>
#include <stdio.h>
#include <string.h>

if (fd == -1) {
    fprintf(stderr, "open failed: %s\n", strerror(errno));
}

perror is a convenient alternative:

if (fd == -1) {
    perror("open");
}

The value of errno is meaningful only after a function reports an error. A successful function call is not required to clear it, so this is wrong:

read(fd, buffer, size);
if (errno != 0) {     // wrong: errno may be left over from an earlier call
    // ...
}

Always inspect the function's documented return value first. The Linux manual's errno(3) page makes this rule explicit.

Common errors include:

Do not memorise this table as exhaustive. Use man 2 open, man 2 read or the relevant manual page for the call being made.

Common I/O Types#

The function signatures deliberately use types suited to byte counts and file offsets:

read cannot return a size_t, because an unsigned type has no ordinary -1 error value. ssize_t can represent both non-negative byte counts and failure.

Avoid storing a read result directly in size_t:

size_t n = read(fd, buffer, sizeof buffer);  // dangerous

If read returns -1, converting it to unsigned produces a very large number. Use ssize_t and test for -1 before any conversion.

Worked Example: Copying a File#

This example combines open, read, repeated write and close:

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv) {
    if (argc != 3) {
        fprintf(stderr, "usage: %s <source> <destination>\n", argv[0]);
        return EXIT_FAILURE;
    }

    int input_fd = open(argv[1], O_RDONLY);
    if (input_fd == -1) {
        perror(argv[1]);
        return EXIT_FAILURE;
    }

    int output_fd = open(
        argv[2],
        O_WRONLY | O_CREAT | O_TRUNC,
        0644
    );
    if (output_fd == -1) {
        perror(argv[2]);
        close(input_fd);
        return EXIT_FAILURE;
    }

    unsigned char buffer[4096];
    int failed = 0;

    while (!failed) {
        ssize_t n_read = read(input_fd, buffer, sizeof buffer);

        if (n_read == 0) {
            break;
        }
        if (n_read == -1) {
            if (errno == EINTR) {
                continue;
            }
            perror("read");
            failed = 1;
            break;
        }

        ssize_t offset = 0;
        while (offset < n_read) {
            ssize_t n_written = write(
                output_fd,
                buffer + offset,
                (size_t)(n_read - offset)
            );

            if (n_written == -1) {
                if (errno == EINTR) {
                    continue;
                }
                perror("write");
                failed = 1;
                break;
            }

            if (n_written == 0) {
                fprintf(stderr, "write made no progress\n");
                failed = 1;
                break;
            }

            offset += n_written;
        }
    }

    if (close(input_fd) == -1) {
        perror("close input");
        failed = 1;
    }
    if (close(output_fd) == -1) {
        perror("close output");
        failed = 1;
    }

    return failed ? EXIT_FAILURE : EXIT_SUCCESS;
}

The important mechanics are:

• read continues until it returns 0;
• the buffer is treated as bytes, so it can copy text or binary data;
• partial writes are handled by an inner loop;
• EINTR retries the interrupted operation;
• each opened descriptor is eventually closed;
• every operation's return value is checked.

File Descriptors Versus FILE *#

Low-level POSIX I/O uses integer descriptors, while the C standard I/O library uses buffered FILE * streams. A stream on a Unix-like system normally refers to an underlying descriptor but also maintains its own buffer, positioning and status information. Files and Streams gives the complete comparison and explains how to coordinate the two interfaces safely.

Common Mistakes#

• Treating a file descriptor as a pointer or the file's contents. It is a small process-local handle.
• Assuming the descriptor number identifies the same file across processes.
• Forgetting that read and write may transfer fewer bytes than requested.
• Treating read data as a C string without adding a null terminator.
• Using strlen for arbitrary file bytes. Binary data may contain zeroes.
• Storing an ssize_t result in size_t before checking for -1.
• Supplying O_CREAT without the mode argument.
• Passing 644 instead of octal 0644 for permissions.
• Using errno without first observing an error return.
• Forgetting to close descriptors on error paths.
• Assuming O_APPEND is identical to calling lseek once. Append mode positions each write at the end as part of the write operation.
• Assuming a successful write means the bytes have already reached durable physical storage.

Practice#

1. What is the difference between a pathname, a file descriptor and an open file description?
2. Why does read return ssize_t rather than size_t?
3. What does read returning zero mean for a regular file?
4. Explain O_WRONLY | O_CREAT | O_TRUNC.
5. Why is 0644 written with a leading zero?
6. A call asks write to transfer 100 bytes and returns 37. Did it fail?
7. Why should diagnostics normally go to descriptor 2 rather than descriptor 1?

These operations are concrete examples of the controlled service boundary introduced in Operating Systems. Files and Streams continues from these low-level descriptor operations into buffered standard I/O.