Skip to content

The Markdown for agents drops the Arabic

markdown-arabic-lossminor
On this page

Why it matters

  • Some sites, or the services in front of them, give AI agents a page as Markdown when they ask for it with Accept: text/markdown, so they read the text without the markup.
  • Converters can drop right-to-left text, write it in the wrong encoding, or keep only the menus and footer. The agent then sees another page than the visitor does.
  • What the agent reads is what it quotes in its answers: Arabic lost from the Markdown is lost from the answers.

How to fix

  • Open the page as an agent asks for it, and compare the text with what the browser shows:
text
curl -H "Accept: text/markdown" https://example.com/page
  • If a service in front of the site converts it, make sure the page is sent as UTF-8 and that its main text is in content elements (main, article), not in elements the converter strips.
  • If you produce the Markdown yourself, produce it from the content's source, not from the page's HTML after scripts build it.

How we detect

We ask for the page itself once with Accept: text/markdown, where robots.txt allows. When it answers text/markdown, and the page is mostly Arabic with at least 30 distinct Arabic words, we compare the Markdown's words with the page's, after folding hamza forms, ta marbuta and alef maqsura and removing diacritics. The rule fails when the Markdown keeps under half of the page's distinct words, or no Arabic letter at all.