AI Forum in the field of Robust Language Models

Michal Štefánik, an NLP Researcher at the University of Helsinki, gave a lecture as part of the Slovak NLP community event on Robust Language Models and Where to Find Them.

Language models (LMs) have emerged as a technology adopted in a wide variety of use-cases, today largely exceeding traditional NLP tasks. Despite that, over the last years, we have made little progress in LMs’ applicability in tasks with limited data and tasks requiring reliable decision making, bearing huge potential for automation. Both these limitations can be attributed to models’ limited robustness in out-of-distribution settings. In this talk, I will share our experience with making language models more robust across languages, domains and tasks, and underline some general rules for improving robustness. Finally, I will outline our vision for achieving progress beyond the inherent limitations of the Transformer architecture, motivating further research in several key directions.