16. Unicode and UTF-8

ASCII#

ASCII is a fixed-width 7-bit encoding with 128 values, 0x00 through 0x7F. It includes:

• control characters such as NUL ('\0') and line feed ('\n');
• punctuation;
• digits '0' through '9';
• uppercase 'A' through 'Z';
• lowercase 'a' through 'z'.

The useful blocks are sequential:

int digit = c - '0';          // if c is known to be '0' ... '9'
char next = letter + 1;       // 'a' becomes 'b'

The digit trick works because the code points for the ten digits are consecutive. Do not infer that arbitrary character arithmetic is meaningful: punctuation also occupies gaps between the alphabetic blocks.

Unicode#

Unicode provides a common coded character set for writing systems, symbols and control characters. Its codespace is simply the full range of numbers reserved for possible code points: U+0000 through U+10FFFF, giving 1,114,112 positions. Not every position is assigned to a character.

A Unicode scalar value is a code point which an encoding such as UTF-8 is allowed to represent: any value in that codespace except U+D800 through U+DFFF. That excluded surrogate range exists for UTF-16's internal design; it is not a set of standalone characters.

Code points are conventionally written as U+ followed by at least four hexadecimal digits:

Unicode is not merely “one giant font”. It standardises character identities and properties. A glyph is the visible shape eventually drawn by a particular font and rendering system. One character can have different glyphs in different fonts, and one displayed glyph can sometimes depend on several code points.

UTF-32#

UTF-32 uses a 32-bit code unit for each Unicode scalar value. Conceptually, the value stored is the code point:

A   U+0041   -> 00000041
€   U+20AC   -> 000020AC
字  U+5B57   -> 00005B57
😀  U+1F600  -> 0001F600

The fixed width makes indexing by code point simple, but wastes space for ASCII-heavy text. It also still does not guarantee that one code unit equals one user-perceived character.

UTF-8#

UTF-8 is a variable-width encoding which represents one Unicode scalar value using one to four bytes:

The x bits carry the code point; the fixed 0, 10, 110, 1110 and 11110 bits describe the structure. A continuation byte reserves its first two bits as 10, leaving six of its eight bits available for code-point data. Prefixes make the structure self-identifying:

• 0xxxxxxx is an ASCII byte and a complete one-byte character;
• 110xxxxx, 1110xxxx and 11110xxx begin two-, three- and four-byte sequences;
• 10xxxxxx is a continuation byte.

Use the leading-byte table and image above as a decision procedure: 0xxxxxxx means one byte total, 110xxxxx means two, 1110xxxx means three, and 11110xxx means four. A byte beginning 10 is a continuation byte, so it cannot begin a new character by itself.

ASCII is therefore a byte-for-byte subset of UTF-8. Bytes 0x00 through 0x7F never appear inside the multibyte encoding of another code point. In particular, a continuation byte cannot be zero or /, which lets UTF-8 fit safely into the existing Unix conventions for NUL-terminated pathnames.

Practice: Quick Checks#

1. Distinguish a character, a code point, a Unicode scalar value and a UTF-8 byte sequence.
2. What prefix identifies a continuation byte, and which masks can test its two high bits?

Encoding a Code Point#

Take €, code point U+20AC:

1. Write the value in binary: 10 000010 101100.
2. Split it from the right into six-bit groups.
3. It needs three groups, so choose the three-byte template.
4. Insert the payload bits and zero-fill unused leading payload positions.

template: 1110xxxx 10xxxxxx 10xxxxxx
payload:         10   000010   101100
result:   11100010 10000010 10101100
hex:         E2       82       AC

More examples:

Decoding UTF-8#

To recover the code point:

1. determine the length from the leading-byte prefix;
2. verify that the required following bytes begin with 10;
3. remove all structural prefix bits;
4. concatenate the payload bits.

For E2 82 AC:

11100010 10000010 10101100
    0010   000010   101100
       10 000010 101100
       0x20AC

Validating real UTF-8 requires more than checking prefixes. A decoder must reject:

• truncated sequences;
• a continuation byte where a leading byte is expected;
• a non-continuation byte inside a multibyte sequence;
• overlong encodings, where a value uses more bytes than necessary;
• surrogate code points U+D800 through U+DFFF;
• values above U+10FFFF.

Practice: Encoding, Decoding and Validation#

3. Decode F0 9F 98 80 by removing the structural prefix bits and concatenating the payload. Which scalar value is it?
4. Explain why each sequence is invalid UTF-8: E2 82, C0 AF, ED A0 80, and F4 90 80 80.

Working with UTF-8 in C#

A C string is still a zero-terminated array of char. The library sees bytes; it does not automatically see Unicode code points.

char message[] = "A€字😀";
printf("%zu bytes\n", strlen(message));

In UTF-8 this contains 1 + 3 + 3 + 4 = 11 bytes before the terminating zero. strlen(message) therefore returns 11, not 4.

If the input is already known to be well-formed UTF-8, a simple code-point count can count bytes which are not continuation bytes:

#include <stddef.h>

size_t utf8_code_points(const unsigned char *s) {
	size_t count = 0;

	for (size_t i = 0; s[i] != '\0'; i++) {
		if ((s[i] & 0xC0) != 0x80) {
			count++;
		}
	}

	return count;
}

0xC0 is 11000000. Masking with it preserves the first two bits. Continuation bytes have the pattern 10xxxxxx, so they produce 0x80 and are not counted. Bitwise Operations develops the mask-and-compare technique in detail.

This is not a validator. Invalid leading bytes would also be counted. A safer routine validates and advances according to the leading byte:

int utf8_sequence_length(unsigned char byte) {
	if ((byte & 0x80) == 0x00) return 1; // 0xxxxxxx
	if ((byte & 0xE0) == 0xC0) return 2; // 110xxxxx
	if ((byte & 0xF0) == 0xE0) return 3; // 1110xxxx
	if ((byte & 0xF8) == 0xF0) return 4; // 11110xxx
	return -1;
}

The full decoder must then check each continuation byte and reject illegal values.

UTF-8 Across Read Boundaries#

A file or pipe returns chunks of bytes, not whole Unicode characters. A valid multibyte sequence can therefore be split between two successful reads:

Suppose the stream is 41 E2 82 AC 0A, which encodes A, €, then a newline. A first read might return only 41 E2 82; the second then returns AC 0A.

After the first read, 41 is a complete ASCII code point, but E2 82 is only the beginning of the three-byte encoding E2 82 AC. It is not invalid merely because the current buffer ended. A streaming decoder retains that incomplete prefix and completes it when the next chunk arrives. It reports truncation only if the overall input ends before the required continuation bytes arrive.

This separates two different events: end of the current buffer means “obtain more bytes”, while end of the entire input means an unfinished sequence is truly truncated.

Practice: Writing and Debugging#

5. The simple utf8_sequence_length() helper accepts leading byte 0xC0 as a two-byte start. Why is that insufficient validation, and what later check is required?
6. A loop counts every byte for which (byte & 0xC0) != 0x80. Give one kind of malformed input it would still count as a code-point start.

Bytes, Code Points and Display Width#

These counts answer different questions:

For plain ASCII, all four often coincide. Unicode makes the distinction visible.

Practice: Challenge#

7. The UTF-8 string A€😀 is followed by a terminating zero. What do strlen() and a valid code-point counter return, and how many bytes of array storage are occupied including the terminator?
8. Why can neither the UTF-8 byte count nor the code-point count reliably predict terminal columns?