Report on the Current State of Societal Biases in Slovak AI

Pikuliak, M., Oreško, Š., Burda, K., Gavorník, A., Mesarčík, M., Hrčková, A., Kottulová, J., Szapuová, M., Podroužek, J., Šimko, M.

This report serves as a concluding report of our project Societal Biases in Slovak AI running from November 2022 to October 2023. The goal of our project was to understand how gender biases in particular affect Slovak AI systems, but also to increase the awareness of this issue in both the expert community and the general public, as this is an issue that affects virtually everyone using modern communication technologies. Our aim was to approach this issue from an interdisciplinary point of view. The team included people with technical expertise in AI and natural language processing; but also AI ethics experts, social scientists, translators, and gender experts.

The main contribution of our project is the evaluation of multiple types of AI systems, namely English-to-Slovak machine translation systems, Slovak masked language models, and Slovak speech-to-text systems. We have proposed and implemented an evaluation methodology and observed whether these systems exhibit biased behavior with respect to gender. Each experiment was aimed at a specific type of biased behavior, such as male as norm behavior, stereotypical thinking, or non-equitable performance. Worryingly, we were able to discover some sort of problematic behavior in all the models taken into consideration

Cite: Pikuliak, M., Oreško, Š., Burda, K., Gavorník, A., Mesarčík, M., Hrčková, A., Kottulová, J., Szapuová, M., Podroužek, J., & Šimko, M. (2023). Report on the Current State of Societal Biases in Slovak AI. Zenodo. https://doi.org/10.5281/zenodo.10059075.