Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

Šuppa, M.^{1, 2}, Vykopal, I., Kopčan, J.

¹ Comenius University Bratislava, Slovakia
² Cisco Systems

Massively multilingual pipelines such as FineWeb2 apply one recipe to a thousand languages, so no single language’s failure modes are anticipated and the communities who could recognise them have no cheap way to express them. We ask whether an LLM agent can debug a corpus: given read-only tools and a generic instruction, can it find reproducible failure slices and compile them into cheap executable tests? Auditing the 26.5M-document Slovak portion of FineWeb2 on a 300-document inspection budget, it recovered seven slices over four phenomena, including a machine-generated content farm of 1,408,437 documents (5.3%) that no static signal flags. 86.6% of it passes every filter at once. Five of the seven lie almost entirely outside the obvious audit slices, though the largest is common enough that non-agentic baselines propose it too. A stronger model also found the shipped language confidence is not a probability: 40.4% of documents score above 1.0. The suite runs over the corpus in 0.7s without a GPU, against $7.50 for the audit and 12k–$90k for one LLM classification pass. A native speaker labelled 255 blinded documents: 6 of the 7 slices carry defect rates of .53–1.00 against .08 in a random control, with 1 unsupported at .07. After review we release the framework, the Slovak test suite, validated examples and the agent trajectories.

Cite: Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan. Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2.