Definition
- UTF-8 is a transformation format of the universal character set (Unicode and ISO 10646). It writes each character from
U+0000toU+10FFFFas one to four bytes, and ASCII characters stay a single byte with their usual value (RFC 3629, sections 1 and 3). - Characters from
U+0080toU+07FFtake two bytes (RFC 3629, section 3). The Arabic letter ق (U+0642) falls in that range, so it is written as the two bytesD9 82, as is every letter in the basic Arabic block. - The WHATWG Encoding Standard calls UTF-8 the most appropriate encoding for interchange of Unicode, and says new protocols and formats must use it exclusively. HTML (section 4.2.5.4) requires the document's actual encoding to be UTF-8, and any declaration of it to use the label
utf-8, in any letter case. - The encoding is declared in the server's
Content-Type: text/html; charset=utf-8header, or in a<meta charset="utf-8">tag that must fit entirely within the first 1024 bytes of the document. When the two disagree, the server's header wins, and a byte order mark (BOM) at the start of the file outranks both (W3C).
Why it matters
- UTF-8 carries every Unicode character, so Arabic, English, digits and symbols can share one page without a legacy encoding for each language.
- When UTF-8 bytes are read as another encoding, the text comes out garbled: every Arabic letter is two bytes, which Windows-1252 reads as two Latin characters. That is mojibake, which the
ar-mojibakerule and the Arabic encoding checker detect. - A declaration alone does not repair a file: the W3C says you also have to save the file as UTF-8.
- HTML notes that encodings other than UTF-8 cause problems with form submission and URL encoding, because both use the document's encoding by default. That is how UTF-8 ties into percent-encoding in links.
- The encoding does not tell the browser what language the text is in: the
langattribute does that, and neither replaces the other.
Example
The declaration in the server's header and at the top of <head>, with the file itself saved as UTF-8:
Content-Type: text/html; charset=utf-8html
<!doctype html>
<html lang="ar" dir="rtl">
<head>
<meta charset="utf-8" />
<title>المتجر</title>
</head>
</html>This is how the letter ق is written as bytes:
js
new TextEncoder().encode('ق') // Uint8Array [217, 130], that is D9 82Common mistakes
- Declaring UTF-8 without saving the file in it: the declaration has to match the bytes.
- Putting the
<meta charset>tag after the first 1024 bytes of the file, for instance behind long inline scripts or styles. Place it right after<head>. - Letting the header and the tag disagree: the header wins, so the tag misleads whoever reads the code.
- Using both a
charsettag and anhttp-equivtag for the encoding: HTML does not allow the two in one document. - Leaving the database, or its connection, in another encoding, so text goes in and comes out garbled. The
ar-mojibakerule suggests theutf8mb4encoding in MySQL.