How UTF-8 stores text
UTF-8 is the most common way the web turns text into bytes. The idea to keep: the first byte of a character says how many bytes that character takes.
The short version
Unicode gives every character a number, its code point; 日 is U+65E5. UTF-8 stores that number in one to four bytes, fewer for smaller numbers. Plain ASCII text is already valid UTF-8. For bigger numbers, the first byte starts with 1s that count the bytes in the character, and every later byte starts with 10.
Step by step
- Take the code point: 日 is U+65E5.
- Find its row in the table below: it is a three-byte one.
- Write the number in binary and fill the
xslots from left to right:0110,010111,100101. - The result is E6 97 A5, as the diagram shows.
| Code point range | Bytes | Bit pattern |
|---|---|---|
| U+0000–U+007F | 1 | 0xxxxxxx |
| U+0080–U+07FF | 2 | 110xxxxx 10xxxxxx |
| U+0800–U+FFFF | 3 | 1110xxxx 10xxxxxx 10xxxxxx |
| U+10000–U+10FFFF | 4 | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
Why it matters
Continuation bytes are always marked, so a reader dropped into the middle of a stream can find where the next character starts. The upside is ASCII compatibility and one encoding for every script. The downside is that characters differ in length. My reading: you cannot jump straight to the Nth character, you have to count from the start. Cut a string between bytes and the character breaks, as MDN warns.
Sources
- RFC 3629: UTF-8, a transformation format of ISO 10646 — the byte table, ASCII as valid UTF-8, the first byte giving the length, continuation bytes starting with 10, the U+65E5 example
- UTF-8 (MDN Glossary) — one to four bytes per character, the most common encoding on the web, ASCII compatibility, bytes corrupted when split and rejoined wrongly