Research areas: Parameter-Efficient Fine-Tuning, Multi-Task Transfer-Learning, Model Merging, Efficient and Low-Resource AI, Natural Language Processing
Position: PhD Student
Robert is a PhD student focusing on parameter-efficient multi-task transfer learning with limited labeled data in natural language processing. In addition, he is also interested in the generalizability of AI models and model merging methods.
He holds a master’s degree from the Faculty of Mathematics, Physics and Informatics, Comenius University Bratislava, where he graduated with honors. Both his bachelor’s (Realistic image synthesis, based on a description in natural language) and master’s (Controlling the output of a generative model by finding latent feature representations) theses were related to generative models (mostly GANs for image generation) and the methods of controlling their output without further training.
During his studies, he also gained 3 years of experience in DevOps and operating systems at a Slovak web hosting company (Websupport).
With the increasing popularity of Large Language Models (LLMs) based on the transformer architecture, the number of trainable parameters has steadily increased. LLMs currently contain billions of trainable parameters, making them power and cost-inefficient. LLMs also require significant training data, primarily benefiting well-resourced languages. Parameter-efficient fine-tuning (PEFT) methods have emerged to address these problems. PEFT methods aim to fine-tune pre-trained language models with only a fraction of their parameters. Multi-task PEFT methods further extend the original PEFT idea by leveraging multi-task transfer learning to solve multiple tasks. This gives us the benefit of having only a single expert model per multiple tasks.
In his dissertation, he aims to improve multi-task PEFT methods and to explore the factors that affect their efficiency and performance. Current multi-task PEFT methods often rely on a strict selection of source tasks and, in most cases, do not investigate how different selections affect the results. Additionally, the sensitivity of PEFT methods to various multitask learning factors remains underexplored.
Supervising team: Mária Bieliková (KInIT), Ivan Srba (KInIT)
Reviewer at conferences:
Academic self-governance engagement:
