SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

Šuppa, M.^{1, 2}, Ridzik, A., Hládek, D.³, Kňažeková, N., Ondrejová, V.²

¹ Comenius University Bratislava, Slovakia
² Cisco Systems
³ Technical University of Košice, Slovakia

We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types—nearly 4× the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop e5-sk-small (45M parameters) and e5-sk-large (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62%, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages.

Cite: Marek Šuppa, Andrej Ridzik, Daniel Hládek, Natália Kňažeková, and Viktória Ondrejová. SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 45597–45628, San Diego, California, United States. Association for Computational Linguistics. 2026.