{"id":28614,"date":"2023-05-19T11:24:36","date_gmt":"2023-05-19T09:24:36","guid":{"rendered":"https:\/\/kinit.sk\/publication\/in-depth-look-at-word-filling-societal-bias-measures\/"},"modified":"2026-04-23T15:09:16","modified_gmt":"2026-04-23T13:09:16","slug":"in-depth-look-at-word-filling-societal-bias-measures","status":"publish","type":"publication","link":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/","title":{"rendered":"In-Depth Look at Word Filling Societal Bias Measures"},"content":{"rendered":"<div id=\"\" class=\"element core-paragraph\">\n<p><strong>Pikuliak, M., Be\u0148ov\u00e1, I., Bachrat\u00fd, V.<\/strong><\/p>\n<\/div>\n\n<div id=\"\" class=\"element core-paragraph\">\n<p>Many measures of societal bias in language models have been proposed in recent years. A popular approach is to use a set of word filling prompts to evaluate the behavior of the language models. In this work, we analyze the validity of two such measures \u2013 StereoSet and CrowS-Pairs. We show that these measures produce unexpected and illogical results when appropriate control group samples are constructed. Based on this, we believe that they are problematic and using them in the future should be reconsidered. We propose a way forward with an improved testing protocol. Finally, we also introduce a new gender bias dataset for Slovak.<\/p>\n<\/div>\n\n<div id=\"\" class=\"element core-paragraph\">\n<p>Cite: Pikuliak, M., Be\u0148ov\u00e1, I., Bachrat\u00fd, V. In-Depth Look at Word Filling Societal Bias Measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3648\u20133665, Dubrovnik, Croatia. Association for Computational Linguistics (2023). DOI: <a href=\"https:\/\/doi.org\/10.18653\/v1\/2023.eacl-main.265\">10.18653\/v1\/2023.eacl-main.265<\/a>.<\/p>\n<\/div>","protected":false},"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"categories":[76,142],"class_list":["post-28614","publication","type-publication","status-publish","hentry","category-natural-language-processing-sk","category-2023-sk"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>In-Depth Look at Word Filling Societal Bias Measures - KInIT<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/\" \/>\n<meta property=\"og:locale\" content=\"sk_SK\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"In-Depth Look at Word Filling Societal Bias Measures - KInIT\" \/>\n<meta property=\"og:description\" content=\"Pikuliak, M., Be\u0148ov\u00e1, I., Bachrat\u00fd, V. Many measures of societal bias in language models have been proposed in recent years. A popular approach is to use a set of word...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/\" \/>\n<meta property=\"og:site_name\" content=\"KInIT\" \/>\n<meta property=\"article:modified_time\" content=\"2026-04-23T13:09:16+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/kinit.sk\/wp-content\/uploads\/2021\/03\/KINIT_Sharepic.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@kinit\" \/>\n<meta name=\"twitter:label1\" content=\"Predpokladan\u00fd \u010das \u010d\u00edtania\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 min\u00fata\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/kinit.sk\\\/sk\\\/publikacia\\\/in-depth-look-at-word-filling-societal-bias-measures\\\/\",\"url\":\"https:\\\/\\\/kinit.sk\\\/sk\\\/publikacia\\\/in-depth-look-at-word-filling-societal-bias-measures\\\/\",\"name\":\"In-Depth Look at Word Filling Societal Bias Measures - KInIT\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/kinit.sk\\\/#website\"},\"datePublished\":\"2023-05-19T09:24:36+00:00\",\"dateModified\":\"2026-04-23T13:09:16+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/kinit.sk\\\/sk\\\/publikacia\\\/in-depth-look-at-word-filling-societal-bias-measures\\\/#breadcrumb\"},\"inLanguage\":\"sk-SK\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/kinit.sk\\\/sk\\\/publikacia\\\/in-depth-look-at-word-filling-societal-bias-measures\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/kinit.sk\\\/sk\\\/publikacia\\\/in-depth-look-at-word-filling-societal-bias-measures\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/kinit.sk\\\/sk\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Natural Language Processing\",\"item\":\"https:\\\/\\\/kinit.sk\\\/sk\\\/category\\\/natural-language-processing-sk\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"In-Depth Look at Word Filling Societal Bias Measures\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/kinit.sk\\\/#website\",\"url\":\"https:\\\/\\\/kinit.sk\\\/\",\"name\":\"KInIT\",\"description\":\"Vyu\u017e\u00edvame v\u00fdskum pre \u013eud\u00ed a priemysel\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/kinit.sk\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"sk-SK\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"In-Depth Look at Word Filling Societal Bias Measures - KInIT","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/","og_locale":"sk_SK","og_type":"article","og_title":"In-Depth Look at Word Filling Societal Bias Measures - KInIT","og_description":"Pikuliak, M., Be\u0148ov\u00e1, I., Bachrat\u00fd, V. Many measures of societal bias in language models have been proposed in recent years. A popular approach is to use a set of word...","og_url":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/","og_site_name":"KInIT","article_modified_time":"2026-04-23T13:09:16+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/kinit.sk\/wp-content\/uploads\/2021\/03\/KINIT_Sharepic.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@kinit","twitter_misc":{"Predpokladan\u00fd \u010das \u010d\u00edtania":"1 min\u00fata"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/","url":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/","name":"In-Depth Look at Word Filling Societal Bias Measures - KInIT","isPartOf":{"@id":"https:\/\/kinit.sk\/#website"},"datePublished":"2023-05-19T09:24:36+00:00","dateModified":"2026-04-23T13:09:16+00:00","breadcrumb":{"@id":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/#breadcrumb"},"inLanguage":"sk-SK","potentialAction":[{"@type":"ReadAction","target":["https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/kinit.sk\/sk\/publikacia\/in-depth-look-at-word-filling-societal-bias-measures\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/kinit.sk\/sk\/"},{"@type":"ListItem","position":2,"name":"Natural Language Processing","item":"https:\/\/kinit.sk\/sk\/category\/natural-language-processing-sk\/"},{"@type":"ListItem","position":3,"name":"In-Depth Look at Word Filling Societal Bias Measures"}]},{"@type":"WebSite","@id":"https:\/\/kinit.sk\/#website","url":"https:\/\/kinit.sk\/","name":"KInIT","description":"Vyu\u017e\u00edvame v\u00fdskum pre \u013eud\u00ed a priemysel","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/kinit.sk\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"sk-SK"}]}},"_links":{"self":[{"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/publication\/28614","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/types\/publication"}],"version-history":[{"count":2,"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/publication\/28614\/revisions"}],"predecessor-version":[{"id":39659,"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/publication\/28614\/revisions\/39659"}],"wp:attachment":[{"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/media?parent=28614"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kinit.sk\/sk\/wp-json\/wp\/v2\/categories?post=28614"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}