WTF is UTF-8?

01Intro

Have you ever wondered how computers can display all the different languages and symbols used in the world? From the websites you browse to the apps you use, UTF-8 is the invisible backbone that powers multilingual communication in the digital realm. Most of my programming life I kinda ignored it — even after seeing it literally everywhere.

02A Code-Length Conundrum in Go

I have been delving a bit deep into the Golang world recently when I stumbled upon something interesting.

fmt.Println(len("日本語")) // what do you think the length is?

Well I don't know about you, but my initial guess was 3. The result? 9. NINE bytes. Coming from a Python/JavaScript background, I thought one character would mean adding 1 to the length.

To understand this better, I read that a rune is Go's name for a Unicode code point. A rune is not necessarily a complete user-perceived character, though. That distinction led me back to a more basic question: how did computers agree on a way to represent every language in the first place?

03Unicode Code Points

A computer at its core only understands bits, so we need to encode the information we use in our languages into a form that can be represented with them. It was easier when we were only thinking about English letters — ASCII assigns values from 0 to 127, so each value fits in 7 bits, with one bit to spare in an 8-bit byte.

As communication spanned languages and cultures, every region shipped its own encoding scheme — so the same bytes rendered differently depending on which table you happened to be holding. Send an email abroad and the characters arrived scrambled.

The missing piece was a shared catalogue of characters. Unicode provides one: it assigns each character an abstract number called a code point. The letter A, for example, is U+0041. That identity is useful everywhere, but it still does not say how the character is represented in memory.

04UTF-8

Unicode gave us the identities. UTF-8 is the encoding: it translates valid code points into byte sequences, one byte for U+0000–U+007F and two to four for everything above. Variable length, so it only spends extra bytes when it has to. That <meta charset="UTF-8"> you've typed a thousand times is you telling the browser which table to read the bytes with.

In other words, Unicode answers “which character is this?” and UTF-8 answers “which bytes should store it?”. UTF-8 keeps ASCII intact — A is still the single byte 0x41 — while making room for the rest of Unicode. The leading bits of each byte also tell a decoder how many bytes belong to the character, which is what makes a variable-length sequence unambiguous.

05Runes in Go

Back in Go, a string is an immutable sequence of bytes. It is not required to contain valid UTF-8, even though UTF-8 is what we usually put in one. Rob Pike put it plainly:

It’s important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes.

Rob Pike · go.dev/blog/strings

A rune gives Go a name for a Unicode code point. It is useful when the value is text rather than an arbitrary byte, but it is still not quite the same thing as a character a person sees on screen. One visible character can be made from more than one code point.

"Code point" is a bit of a mouthful, so Go introduces a shorter term for the concept: rune. It means exactly the same as "code point", with one interesting addition. Go defines rune as an alias for int32, so a program can be clear when an integer value represents a code point. What you might think of as a character constant is called a rune constant in Go.

Rob Pike · go.dev/blog/strings

06Cracking the Length Code

Now return to the original expression. len counts the bytes in a Go string, so the three Japanese code points each contribute three UTF-8 bytes:

fmt.Println(len("日本語"))

So: three Japanese runes × three bytes each = nine. When you need the number of code points instead of bytes, unicode/utf8 has you covered with RuneCountInString. It walks the UTF-8 sequence and returns three for this string.

import (
    "fmt"
    "unicode/utf8"
)

fmt.Println(utf8.RuneCountInString("日本語")) // 3