Ivana Beňová

Ivana Beňová is a PhD student and member of the NLP team, specializing in multimodal processing, evaluation of vision-language models, natural language processing and grounding. She obtained her Master’s degree in Mathematical Economics, Finance and Modelling from Comenius University in Bratislava, Faculty of Mathematics, Physics and Informatics, where she focused on mathematical modelling, optimization, and data-driven methods. During her studies, she participated in several research and analytical projects, including the CFA Research Challenge and research on transport modelling and data analysis.

Currently, Ivana is a PhD student working on multimodal AI systems, with expected graduation in 2026/2027. Her research focuses on understanding and grounding image-language models, multimodal learning, understanding and grounding of models and trustworthy AI systems. She is particularly interested in how large vision-language models process contextual information and how their predictions can be made more reliable.

Her PhD research investigates the interaction between textual and visual understanding in multimodal AI models, including the grounding of context-dependent language in visual environments. Her work also explores multilingual and multimodal fact-check retrieval, explainability, and evaluation of modern AI systems. As part of her research activities, she collaborated internationally through research projects and visits, including cooperation with the Centre for Research & Technology Hellas (CERTH) in Greece, Utrecht University in the Netherlands or George Mason University in the USA.

Supervising team: Marián Šimko (KInIT), Jana Košecká (George Mason University)

The development in artificial intelligence subfields, such as machine learning, natural language processing (NLP), and computer vision (CV), has increased the interest in finding solutions to problems that combine linguistic and visual data. Solving complex image-language challenges might be a stepping stone for artificial intelligence. The majority of the advancements in image-language modeling, along with advances in CV and NLP, were made possible by the development of deep learning. Different multimodal models were introduced after the self-attention, and self-supervised training was expanded to include visual information.

In our proposal, we summarized types of state-of-the-art image-language models. We focused on their differences, pretraining strategies, advantages, and usability for downstream tasks. As these models are deep learning algorithms, we don’t know the learned relationships between language and vision tokens. Therefore, we define the concept of grounding language in vision. We also studied different probing methods, mainly the frequently used image-text matching, to explore the understanding of linguistic aspects by image-language models.

 After obtaining such an extensive survey of the area of understanding and grounding of image-text models, we identified three open problems that we want to study in the following years:

(1) how to eliminate the identified challenges of the image-text matching probing technique,

(2) the lack of knowledge about how different linguistic aspects are grounded in vision in a pretrained image-language models and

(3) how to improve the language grounding in these models using the knowledge obtained from previous experiments.