Why it matters
- Models learn from filtered text. The public FineWeb-2 corpus, which many models are trained on, drops a page that fails its quality filters before a model sees it. A page that is dropped is not part of what the model learns.
- Repetition drops a page more than anything else: the same paragraph on every page, lists of links, or repeated footer text.
- Text that is good for people is good for machines: whole sentences with full stops and punctuation, common Arabic words, and lines that mean something.
Example
<p>نحمّص القهوة الطازجة كل يوم ونشحنها إلى جميع المدن بأفضل الأسعار</p> <p>نحمّص القهوة الطازجة كل يوم ونشحنها إلى جميع المدن بأفضل الأسعار</p> <p>نحمّص القهوة الطازجة كل يوم ونشحنها إلى جميع المدن بأفضل الأسعار</p>
<p>الماء عنصر لا يقل أهمية عن الحبوب نفسها، فالماء الثقيل بالأملاح يخفي النكهات الرقيقة. يُفضَّل أن تكون حرارته قريبة من الخامسة والتسعين مئوية، وأن يُسكب ببطء حتى تتشبع الحبوب بالتساوي.</p>
How to fix
- Write paragraphs of whole sentences that end in a full stop, and have fewer headings and short lists with no explanation.
- Do not repeat the same paragraph on a page or on every page of the site; move shared text to one page.
- Do not stuff the page with lists of links or runs of keywords.
- Keep numbers and symbols next to text that explains them.
How we detect
- We take the visible text of an Arabic page without the navigation, header and footer, a line for each block, and run the filters of FineWeb-2's Arabic (
arb_Arab) configuration in its order: Gopher repetition, FineWeb quality, then Gopher quality. - The thresholds are the values published in
configs/arb_Arab.ymlof huggingface/fineweb-2 (commit d0defb24f193), and what that file does not name comes from the defaults of the datatrove library (commit 1977fbb0f3c1). We invented no number. - FineWeb-2 identifies language with the GlotLID model (fastText), which cannot run here, so we estimate it as the share of the text's letters that are Arabic, held to the same 0.711 threshold, and show it as information rather than a verdict. That estimate cannot tell Arabic from Persian or Urdu. The C4 filters are not in the FineWeb-2 pipeline, so they are shown as a reference, not a verdict.
- Words are split the way spaCy's Arabic tokenizer does, approximately: a run of letters and digits, and each mark by itself. The text here is the page's visible text, not Trafilatura's output.
- Each kind of measure that failed is reported once, with the first and the number that failed. The rule needs 50 words, as the pipeline does. It is a minor finding.