sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak

Šuppa, M.^{1, 2}, Vykopal, I., Ridzik, A., Sopkovič, K.³, Kňažeková, N., Kopčan, J., Blšták, M., Ondrejová, V.², Hládek, D.³, Gregor, M., Tamajka, M., Šimko, M.

¹ Comenius University Bratislava, Slovakia
² Cisco Systems
³ Technical University of Košice, Slovakia

Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation. We present sk-bench, a native-first Slovak benchmark with 30 datasets (33 scored task variants) across ten skill categories. Eleven resources are introduced or first packaged for generative-LLM evaluation, including IFEval-SK with Slovak-adapted instruction checkers and native Chiby/SKJ1 resources for Slovak grammar and morphology. We evaluate 55 open- and closed-weights models under one harness. The best open model trails proprietary APIs by 12.6 points. Model rankings are similar for native and translated closed-form data (ρ≥0.98), though translation separates the strongest models less well. By contrast, human-authored and LLM-generated QA questions rank models differently (ρ=0.72). For Qwen3-14B, continued Slovak pretraining lowers the overall score by 13.9 points. A small instruction set restores three quarters of that loss. Test-time reasoning improves scores by 8.5 to 12.5 points for models of 9B and above. Together, these findings suggest four design lessons for other under-resourced languages: use native data where translation fails, plan instruction repair after language adaptation, enable test-time reasoning before scaling up, and avoid overinvesting in target-language prompts. We release the data and code at https://github.com/slovak-nlp/sk-bench.

Cite: Marek Šuppa, Ivan Vykopal, Andrej Ridzik, Kristián Sopkovič, Natália Kňažeková, Jaroslav Kopcan, Miroslav Blstak, Viktória Ondrejová, Daniel Hládek, Michal Gregor, Martin Tamajka, and Marian Simko. 2026. sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 60008–60055, Budapest, Hungary. Association for Computational Linguistics.