16. Unicode and UTF-8

Unicode and UTF-8#

Computers store bytes, not letters. If we want a byte sequence to represent text, we need an agreement which maps between numbers and the characters a human intends to read. That agreement is a character encoding.

Three ideas must be kept separate:

A code point is not "the bytes". The same code point can be represented using UTF-8, UTF-16 or UTF-32, producing different code units in memory or a file. Keep the fixed-width representations from Integers in mind: an encoding supplies the context which gives each bit pattern meaning.

ASCII#

ASCII is a fixed-width 7-bit encoding with 128 values, 0x00 through 0x7F. It includes:

• control characters such as NUL ('\0') and line feed ('\n');
• punctuation;
• digits '0' through '9';
• uppercase 'A' through 'Z';
• lowercase 'a' through 'z'.

The useful blocks are sequential:

int digit = c - '0';          // if c is known to be '0' ... '9'
char next = letter + 1;       // 'a' becomes 'b'

The digit trick works because the code points for the ten digits are consecutive. Do not infer that arbitrary character arithmetic is meaningful: punctuation also occupies gaps between the alphabetic blocks.

Unicode#

Unicode provides a common coded character set for writing systems, symbols and control characters. Its codespace runs from U+0000 through U+10FFFF, giving 1,114,112 possible code points. Not every code point is assigned to a character, and the surrogate range is not made of Unicode scalar values.

Code points are conventionally written as U+ followed by at least four hexadecimal digits:

Unicode is not merely "one giant font". It standardises character identities and properties. The font and rendering system later decide what glyphs to draw.

UTF-32#

UTF-32 uses a 32-bit code unit for each Unicode scalar value. Conceptually, the value stored is the code point:

A   U+0041   -> 00000041
€   U+20AC   -> 000020AC
字  U+5B57   -> 00005B57
😀  U+1F600  -> 0001F600

The fixed width makes indexing by code point simple, but wastes space for ASCII-heavy text. It also still does not guarantee that one code unit equals one user-perceived character.

UTF-8#

UTF-8 is a variable-width encoding which represents one Unicode scalar value using one to four bytes:

The x bits carry the code point. Prefixes make the structure self-identifying:

• 0xxxxxxx is an ASCII byte and a complete one-byte character;
• 110xxxxx, 1110xxxx and 11110xxx begin two-, three- and four-byte sequences;
• 10xxxxxx is a continuation byte.

flowchart TD
    B["Inspect leading byte"] --> A{"highest bit is 0?"}
    A -->|yes| ONE["1 byte total"]
    A -->|no| P{"prefix"}
    P -->|110| TWO["2 bytes total"]
    P -->|1110| THREE["3 bytes total"]
    P -->|11110| FOUR["4 bytes total"]
    P -->|10 or invalid prefix| BAD["not a valid leading byte"]

ASCII is therefore a byte-for-byte subset of UTF-8. Bytes 0x00 through 0x7F never appear inside the multibyte encoding of another code point. In particular, a continuation byte cannot be zero or /, which lets UTF-8 fit safely into the existing Unix conventions for NUL-terminated pathnames.

Encoding a Code Point#

Take €, code point U+20AC:

1. Write the value in binary: 10 000010 101100.
2. Split it from the right into six-bit groups.
3. It needs three groups, so choose the three-byte template.
4. Insert the payload bits and zero-fill unused leading payload positions.

template: 1110xxxx 10xxxxxx 10xxxxxx
payload:         10   000010   101100
result:   11100010 10000010 10101100
hex:         E2       82       AC

More examples:

Decoding UTF-8#

To recover the code point:

1. determine the length from the leading-byte prefix;
2. verify that the required following bytes begin with 10;
3. remove all structural prefix bits;
4. concatenate the payload bits.

For E2 82 AC:

11100010 10000010 10101100
    0010   000010   101100
       10 000010 101100
       0x20AC

Validating real UTF-8 requires more than checking prefixes. A decoder must reject:

• truncated sequences;
• a continuation byte where a leading byte is expected;
• a non-continuation byte inside a multibyte sequence;
• overlong encodings, where a value uses more bytes than necessary;
• surrogate code points U+D800 through U+DFFF;
• values above U+10FFFF.

Working with UTF-8 in C#

A C string is still a zero-terminated array of char. The library sees bytes; it does not automatically see Unicode code points.

char message[] = "A€字😀";
printf("%zu bytes\n", strlen(message));

In UTF-8 this contains 1 + 3 + 3 + 4 = 11 bytes before the terminating zero. strlen(message) therefore returns 11, not 4.

A simple count of well-formed code-point starts can ignore continuation bytes:

#include <stddef.h>

size_t utf8_code_points(const unsigned char *s) {
	size_t count = 0;

	for (size_t i = 0; s[i] != '\0'; i++) {
		if ((s[i] & 0xC0) != 0x80) {
			count++;
		}
	}

	return count;
}

0xC0 is 11000000. Masking with it preserves the first two bits. Continuation bytes have the pattern 10xxxxxx, so they produce 0x80 and are not counted. Bitwise Operations develops the mask-and-compare technique in detail.

This is not a validator. Invalid leading bytes would also be counted. A safer routine validates and advances according to the leading byte:

int utf8_sequence_length(unsigned char byte) {
	if ((byte & 0x80) == 0x00) return 1; // 0xxxxxxx
	if ((byte & 0xE0) == 0xC0) return 2; // 110xxxxx
	if ((byte & 0xF0) == 0xE0) return 3; // 1110xxxx
	if ((byte & 0xF8) == 0xF0) return 4; // 11110xxx
	return -1;
}

The full decoder must then check each continuation byte and reject illegal values.

Bytes, Code Points and Display Width#

These counts answer different questions:

For plain ASCII, all four often coincide. Unicode makes the distinction visible.

Common Mistakes#

• Calling UTF-8 "Unicode" as though the code-point assignment and byte encoding were the same thing.
• Assuming one char stores one Unicode character.
• Using strlen() to count characters.
• Starting in the middle of a UTF-8 sequence and treating a continuation byte as a complete character.
• Checking only the number of bytes without checking continuation prefixes.

Practice#

1. How many bytes does UTF-8 use to encode code point U+0041 ('A'), and what are its bits?
2. What bit pattern identifies a UTF-8 continuation byte?
3. Why does strlen() fail to return the number of characters in a non-ASCII UTF-8 string?
4. What range of values are Unicode scalar values?
5. How can you determine the length of a UTF-8 sequence just by inspecting the first byte?