Innovative Approach to Use Large Language Models (LLM) for Continues Annotation of Disinformation

Kula, S., Kozik, R.¹, Choras, M.¹

¹ Bydgoszcz University of Science and Technology Bydgoszcz, Poland

Creating solutions that would efficiently, effectively and accurately detect disinformation in online content is currently one of the major challenges for researchers, organizations and communities working on artificial intelligence and cybersecurity solutions. The topic is a current research issue, the importance of which is determined by the constantly growing development of information transmission through various types of Internet platforms used by citizens and societies. Effective methods for detecting disinformation campaigns would better protect societies and democracy. In this article, we focus on annotating content that potentially is disinformation as well as annotating features, that may suggest the existence of disinformation in the content. For annotation purpose, we propose innovative approach to use LLMs (Large Language Models). The goal is to propose effective solution, which imitated the process of annotating content by real human annotators, but does it in objective and fast manner.
In our innovative solution, we replaced people with LLM models and imitated the annotation process conducted by junior and senior annotators, in real-life annotation tasks. In our work, we used the following models: Mistral-Nemo-Instruct-2407 GGUF (imitates a junior annotator), Bielik-11B-v2.3-Instruct
GGUF(imitates a junior annotator) and nemotron-4-340b-instruct (imitates a senior annotator). The content in a language with low resources, i.e., Polish, was analyzed. All experiments and prompts, and also all data were in Polish, but for the purpose of the article, translations of the original prompts and data into English were included. We proved that by using LLMs and the proposed approach it is feasible to obtain the same performance of annotating disinformation content as annotations performed by professional, but costly and slower human annotators.

Cite: Kula, S., Kozik, R., Choraś, M. Innovative Approach to Use Large Language Models (LLM) for Continues Annotation of Disinformation. In IEEE International Conference on Data Mining Workshops, ICDMW 2025, 12-15 November 2025 Washington DC, United States ISBN-13: 979-8-3315-8132-9