Why it matters
- AI assistants and search engines read characters, not shapes. Their tokenizers split text at the characters in it, so tatweel, a zero-width space or a presentation-form letter makes the same word a different one from the word people type in a question.
- Retrieval depends on matching. A passage whose «العروض» has a zero-width space in it is not found by a question that asks for «العروض», and an answer cannot quote it cleanly.
- Numbers and prices are the facts assistants quote. A number in two digit sets (٢٠٢4) is read as two numbers or none.
- Text in pictures is invisible to them. An AI that reads the page sees no sentence where the sentence is a pixel image.
Example
<h1>الـعـروض الـخـاصـة</h1>
<h1>العروض الخاصة</h1>
How to fix
- Tatweel: write the word plainly («العروض»), and stretch headings with CSS if you want that look.
- Invisible characters: delete them. Search the source for U+200B, U+2060 and U+FEFF; keep a direction mark only between words, not inside one.
- Digits: use one digit set in a number: «٢٠٢٤» or «2024», not both.
- Diacritics: keep them for the words that need them to be read right, and write running text plain.
- Presentation forms: type the base letters, or convert the text with Unicode normalization (NFKC) before it is stored:
js
text.normalize('NFKC')- Text in pictures: put the sentences in HTML as text, and keep the alt for what the picture shows.
How we detect
- We read the visible text of the page's HTML on pages in Arabic, and leave out code tags.
- Tatweel: a word with tatweel between two Arabic letters, as the tatweel rule counts it.
- Invisible characters: U+200B, U+2060 or U+FEFF in a word, or a direction mark (U+200E, U+200F, U+061C) between two letters or digits.
- Digits: one number, with its separators, that holds Arabic-Indic or Persian digits together with 0 to 9.
- Diacritics: paragraphs of 20 Arabic letters or more where marks are a quarter of the letters or more, reported when they hold 100 letters in all; paragraphs with Quranic annotation marks are left out.
- Presentation forms: letters in U+FB50 to U+FEFE, leaving out the ornate parentheses and the symbols ﷺ ﷻ ﷼ ﷽.
- Pictures: fewer than 60 Arabic letters in the page's text, and image alt texts that are Arabic sentences of eight words or more.
- We report each kind once, with how many we found and the first one.