09. Operating Systems

A World Without an Operating System#

Without an operating system, every program would need to manage the physical machine itself. Among other things, a program would need to:

• initialise the computer and decide how to load code;
• understand the exact model and protocol of every connected device;
• decide where its data may live in physical memory;
• invent a method for locating persistent data on raw storage;
• coordinate the CPU with every other program;
• prevent other code from reading or overwriting its state;
• implement input, output and networking directly against hardware.

A program written for one hardware configuration would be deeply tied to it. Changing the storage controller, network device or even memory layout could require a rewrite.

The operating system centralises this difficult machinery. Programs target a stable interface supplied by the OS, whilst the OS and its device drivers deal with the hardware-specific details.

What an Operating System Provides#

Abstraction#

The OS presents simpler virtual objects in place of raw hardware:

An abstraction is a useful simplified view which hides machinery you should not need to manage every time. A file, for example, lets a program think in terms of names and ordered bytes. The operating system translates that request into filesystem lookups, device commands and physical storage operations. The hidden machinery still exists; the abstraction gives it a stable interface.

These abstractions are not merely convenient names. They impose useful behaviour and rules. A file has a pathname, permissions and a current offset when opened; none of those ideas exist in a raw sequence of disk sectors.

Resource management#

Many programs may want the CPU, memory or a device simultaneously. The OS decides how these resources are shared:

• the scheduler decides which process runs on a CPU;
• the virtual-memory system gives each process its own address space;
• filesystems coordinate persistent storage;
• drivers coordinate device access;
• permissions determine which operations a user or process may perform.

Protection#

One process should not be able to read another process's passwords or overwrite the kernel merely because it calculated a bad pointer. The operating system works with hardware protection mechanisms to isolate programs and enforce permissions.

Portability#

Two computers can contain different hardware whilst presenting the same operating-system interface. A program using the POSIX read interface does not need a separate implementation for every model of storage device.

This does not make every compiled program universally portable. The executable must still target the CPU architecture, executable format and operating-system ABI described in Running MIPS. The OS interface nevertheless removes a large amount of hardware-specific work from ordinary applications.

Kernel Mode and User Mode#

Protection depends on the CPU providing at least two privilege levels.

Kernel mode#

The operating-system kernel is the trusted core which remains in control of the machine. It runs in privileged mode, meaning the CPU permits it to perform operations which ordinary applications are deliberately forbidden from performing. It can:

• configure memory protection;
• access device-control registers;
• execute privileged CPU instructions;
• respond to interrupts—events which temporarily pause the current instruction stream so the CPU can handle something requiring attention, such as a timer or completed device operation;
• inspect or modify the state of user processes;
• schedule which program runs next.

Because kernel code has authority over the whole machine, a kernel bug can crash or compromise the entire system.

User mode#

Ordinary applications run in a non-privileged mode. They cannot directly access arbitrary physical memory or devices. Their instructions are constrained by protection state configured by the kernel.

A device driver is kernel code which knows how to operate a particular class of hardware. Applications ask for a high-level operation such as reading bytes; the driver translates that request into commands understood by the storage, network or display device.

If user code could simply switch itself into kernel mode, the separation would be useless. Instead, the CPU provides carefully controlled entry points through exceptions, interrupts and system-call instructions.

flowchart TB
    P["User program<br/>number + arguments"] --> C["CPU performs controlled<br/>entry into kernel mode"]
    C --> K["Kernel validates request<br/>and permissions"]
    K --> H["Permitted hardware<br/>or resource operation"]
    H --> R["Kernel prepares<br/>result or error"]
    R --> U["Program resumes<br/>in user mode"]

System Calls#

A system call is a request from a user program for the kernel to perform an operation which the program cannot or should not perform directly.

Typical system calls include:

• reading and writing bytes;
• opening or closing a file;
• creating, replacing or terminating a process;
• allocating or mapping memory;
• communicating over a network;
• obtaining time or filesystem metadata.

A system call is not an ordinary function call, even though a C wrapper may make it look like one. The request crosses a privilege boundary, changes the CPU into kernel mode at a controlled address, validates untrusted arguments and eventually returns to user mode.

Why validation matters#

The kernel must assume that every user-space argument is untrusted. Before writing to a file, for example, it may need to check:

• whether the supplied file descriptor exists—a small integer handle which identifies one of the process's open kernel-managed I/O connections;
• whether it was opened for writing;
• whether the source buffer belongs to the calling process;
• whether the requested byte count is valid;
• whether the device or filesystem can accept the data;
• whether a signal or resource limit interrupts the operation.

A syntactically valid system call can therefore fail. The kernel does not grant a request merely because the caller reached the correct entry point.

Practice: Quick Checks#

1. What is the difference between entering kernel mode for a system call and performing a context switch?
2. A program passes a valid file descriptor but an invalid buffer pointer to write(). Why must the kernel reject the request instead of trusting libc?

Three Layers of I/O in C#

On Linux, an operation such as printing can be requested at several layers.

Calling syscall directly#

The C library exposes a generic syscall function which takes a system-call number and its arguments:

#include <sys/syscall.h>
#include <unistd.h>

char message[] = "Hello!\n";
syscall(SYS_write, 1, message, 7);

This is useful for experimenting or accessing a newly added call without a wrapper. It is otherwise difficult to read, easy to misuse and tied to a particular operating-system interface and architecture.

Using a named C library wrapper#

The C library on a Unix-like system provides POSIX functions whose names and types correspond to system calls. These interfaces are supplied by libc, but functions such as write() are not part of the ISO C standard library:

#include <unistd.h>

char message[] = "Hello!\n";
write(STDOUT_FILENO, message, 7);

The wrapper supplies a readable name, performs the architecture-specific transition and translates kernel error conventions into the C errno interface.

errno is a thread-local integer used by many library wrappers to record an error code after their return value reports failure. <errno.h> provides readable names for those codes, such as EBADF for a bad file descriptor. Test the function's return value first; an old errno value is not evidence that the latest call failed.

Worked trace: one write request#

Suppose write(1, "hi\n", 3) is called on a terminal:

1. The libc wrapper places the descriptor, buffer address and byte count where the platform ABI requires them.
2. A controlled system-call instruction transfers execution to a kernel entry point. The program has not called a terminal device directly.
3. The kernel looks up descriptor 1, checks that it is open for writing, and safely reads up to three bytes from the process's address space.
4. The terminal driver accepts the bytes now or arranges for the process to wait until it can do so.
5. The kernel returns a count or an error through the ABI, and libc presents that as the C return value and errno convention.

If descriptor 1 is closed, step 3 fails. On Linux the wrapper returns -1 and sets errno to EBADF; no terminal write occurs. If it returns 2, only two bytes were accepted and a caller which requires complete output must try the remaining byte. The requested count is a maximum, not a promise.

Using a higher-level library#

The <stdio.h> library provides buffered and formatted I/O:

#include <stdio.h>

printf("Hello!\n");

printf eventually causes bytes to be written, but it also parses a format string, converts values to text and normally buffers output. One library call does not necessarily correspond to one system call.

flowchart TB
    APP["Application"] --> STDIO["stdio<br/>printf, fopen, fread"]
    STDIO --> WRAP["libc wrappers<br/>write, open, read"]
    WRAP --> SYSCALL["system-call boundary"]
    SYSCALL --> KERNEL["kernel"]

Higher layers are generally more portable and convenient. Lower layers provide greater control and expose the mechanics which COMP1521 is trying to teach. File Systems concentrates on the named open, read, write and close wrappers.

Practice: Tracing and Classification#

3. Trace the layers crossed by printf("x"). Why might the call return without a write system call occurring immediately?
4. Classify each operation as normally user-space work or a kernel service: integer addition, formatting %d, opening a pathname, scheduling another process, and copying bytes from a stdio buffer into a user array.

Mipsy System Calls#

Mipsy emulates a MIPS processor and also supplies a tiny teaching environment for system services. A MIPS program places a service number in $v0, places arguments in designated registers and executes syscall:

li $v0, 4          # print_string service
la $a0, message    # address of argument string
syscall

This resembles the structure of a real system call, but the service numbers and behaviours are specific to the mipsy/SPIM-style teaching interface. They are not Linux x86-64 system-call numbers and should not be mixed with them.

The syscall table in Using MIPS is therefore a table for mipsy programs, not a universal MIPS or Linux table.

The ABI Boundary#

An operating-system ABI specifies details such as:

• system-call numbers;
• registers or stack locations used for arguments;
• how errors and return values are represented;
• data structure layouts;
• executable and object-file conventions;
• rules for calling functions across compiled modules.

This is why source code can be recompiled for different systems but an already compiled executable generally cannot simply move between them. Both the processor's instruction set and the OS ABI must match.

Practice: Challenge#

5. Two machines use the same C source and POSIX function names, but one is x86-64 Linux and the other is MIPS Linux. Why can recompilation work while copying the first machine's executable usually does not?
6. Explain why “the syscall number is correct” is insufficient evidence that a request will succeed.

Inspecting System Calls#

On Linux, strace records the system calls made by a process:

strace ./program

To focus on selected calls:

strace -e trace=openat,read,write,close ./program

The output reveals operations performed indirectly by libraries and by program start-up. Even a tiny C program usually makes more system calls than the lines you wrote suggest, because the dynamic loader and runtime must prepare the process.

Reading the Manual#

The Linux manual is divided into numbered sections. The most relevant sections for COMP1521 are:

When names overlap, the section removes ambiguity. man 2 open documents the system-call interface, whilst another section may document a command or unrelated library interface.

Useful habits include:

man 2 open
man 2 read
man 3 perror
man 7 errno

The manual gives the required headers, function signature, return convention, errors and relevant standards. It will be available in the exam, so learning how to locate information is more valuable than memorising every error code.

Practice: Writing and Debugging#

7. What is wrong with calling syscall(SYS_write, 1, message, 7); and then assuming seven bytes were written? State the checks a robust caller needs.
8. Which manual sections would you try for the read system-call wrapper, printf, and an overview of errno conventions?

Practice: Cumulative#

9. A program executes printf("hello"), but strace shows no immediate write. Give a valid explanation and name one event which could later cause the bytes to be written.
10. A process enters the kernel to call getpid() and then resumes immediately. Must a context switch to another process have occurred? Explain the two distinct transitions.

The next note applies this abstraction and protection machinery to Processes. Later, File Systems follows another important OS abstraction—files and file descriptors—from opening a pathname through transferring bytes.