{"id":37260,"date":"2025-08-04T15:46:34","date_gmt":"2025-08-04T13:46:34","guid":{"rendered":"https:\/\/kinit.sk\/fact-check-retrieval-using-causal-llms\/"},"modified":"2025-08-04T15:51:21","modified_gmt":"2025-08-04T13:51:21","slug":"fact-check-retrieval-using-causal-llms","status":"publish","type":"post","link":"https:\/\/kinit.sk\/sk\/fact-check-retrieval-using-causal-llms\/","title":{"rendered":"Fact-Check Retrieval Using Causal LLMs"},"content":{"rendered":"
\n

In the first instalment<\/a> of our 3-part series, we saw how a text embedding model can be used to retrieve existing fact-checks across different languages and help fact-checkers to cut down on duplicate effort. In this second part, we are going to look at how one can plug causal, generative LLMs (large language models) into the pipeline to refine the results further.<\/strong><\/p>\n<\/div>\n\n

\n

Generative LLMs are now widely understood to have the ability to capture richer semantic representations and to better handle contextual nuances in comparison to text embedding models. They owe this to several factors, such as the scale and generality of their pretraining data, their instruction tuning, their (limited) ability to perform reasoning, and even just the sheer number of parameters.<\/p>\n<\/div>\n\n

\n

LLMs Will Sift through Your Retrieved Fact-Checks<\/h3>\n<\/div>\n\n
\n

So how can we incorporate generative LLMs into the retrieval process? Let us recall what we did last time using TEMs (text embedding models). As shown in Fig. 1, the idea was to simply feed both our documents (fact-checks) and our query (the claim we are interested in) into the TEM to get their embedding vectors. We would then compare the embedding vector of our query against those of all the documents and rank the documents by the resulting similarity scores. The most relevant documents should then be at the top of the list. Note that the individual documents and the query are all embedded independently of each other, so the embeddings of all the documents can be precomputed ahead of time and then reused across all searches.<\/p>\n<\/div>\n\n

\n
\"\"<\/figure>\n<\/div><\/div>\n\n
\n

Fig. 1: Document retrieval using text embedding models (TEMs).<\/em><\/p>\n<\/div>\n\n

\n

When incorporating generative LLMs into the process, we are not going to touch the TEM part, as it is still a very efficient way to get the first rough ranking. You can think of this as performing a Google search \u2013 you get a list of results which are more or less relevant already, but then you need to sift through them and figure out which are really useful to you and which, although superficially similar, are not what you were looking for.<\/p>\n<\/div>\n\n

\n

In our new setup, it is this sifting task that is going to be performed by the generative LLM. The LLM is going to sort through a list of top-ranked documents retrieved by the TEM. It will push the most relevant items towards the top of the list, and it will filter some items out altogether if it finds that they are not actually relevant. The pipeline as a whole is illustrated in Fig. 2.<\/p>\n<\/div>\n\n

\n
\"\"<\/figure>\n<\/div><\/div>\n\n
\n

Fig. 2: A retrieval pipeline combining a text embedding model with an LLM.<\/em><\/p>\n<\/div>\n\n

\n

Why LLMs Have an Easier Time Figuring Out Details<\/h3>\n<\/div>\n\n
\n

Note that the TEM and the LLM tackle the task in two very different ways. The TEM is asked to embed every item independently into a fixed-size vector. Consequently, what it really needs to do is take each document and squash all the information that could possibly be relevant into a very compact form \u2013 and all that without having any idea of what the query is going to be about! It is no wonder that the TEM-based ranking can sometimes miss particular details or ignore various nuances.<\/p>\n<\/div>\n\n

\n

The LLM, in contrast, would naturally have an easier time figuring out details. After all, we are not asking it to blindly squash all the content into one small vector and then make do with the information that survived. On the contrary, the LLM attends to the entire query and to all the documents as it works. It can literally weigh the meaning of each particular word and, having seen the query, consider all this information in the context of what the user is looking for just now.<\/p>\n<\/div>\n\n

\n

Prompting Is Crucial!<\/h3>\n<\/div>\n\n
\n

If you work with large language models often, you may not be surprised that it matters how <\/em>you ask the LLM to perform the task \u2013 LLMs can be quite sensitive to what prompt you provide, and a prompt can often even make or break the entire pipeline. We have experimented with several different setups in our accompanying paper (which is now out on arXiv<\/a>):<\/p>\n<\/div>\n\n

\n