How UTF-8 stores text

· 2 min read

#how-it-works#data

UTF-8 is the most common way the web turns text into bytes. The idea to keep: the first byte of a character says how many bytes that character takes.

The short version

Unicode gives every character a number, its code point; 日 is U+65E5. UTF-8 stores that number in one to four bytes, fewer for smaller numbers. Plain ASCII text is already valid UTF-8. For bigger numbers, the first byte starts with 1s that count the bytes in the character, and every later byte starts with 10.

The character 日 becomes three UTF-8 bytes The code point U+65E5 is encoded into three bytes. The first byte is E6, bits 1110 0110, and says the character takes three bytes. The two continuation bytes are 97, bits 10 010111, and A5, bits 10 100101; both start with 10. U+65E5 E6 97 A5 encode 0110 0101 1110 0101 1110 0110 10 010111 10 100101 code point first byte: 3 bytes continuation: 10… continuation: 10… the three bytes: E6 97 A5
Simplified: one character, three bytes; the first byte counts, the others start with 10.

Step by step

  1. Take the code point: 日 is U+65E5.
  2. Find its row in the table below: it is a three-byte one.
  3. Write the number in binary and fill the x slots from left to right: 0110, 010111, 100101.
  4. The result is E6 97 A5, as the diagram shows.
Code point range Bytes Bit pattern
U+0000–U+007F 1 0xxxxxxx
U+0080–U+07FF 2 110xxxxx 10xxxxxx
U+0800–U+FFFF 3 1110xxxx 10xxxxxx 10xxxxxx
U+10000–U+10FFFF 4 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Why it matters

Continuation bytes are always marked, so a reader dropped into the middle of a stream can find where the next character starts. The upside is ASCII compatibility and one encoding for every script. The downside is that characters differ in length. My reading: you cannot jump straight to the Nth character, you have to count from the start. Cut a string between bytes and the character breaks, as MDN warns.

Sources