Suppose you would like to view a document on the computer without having to browse folders continually. Nowadays, websites often employ AI chatbots for this. In practice, however, these are particularly unintelligent. They cannot answer simple questions or get themselves stuck in a loop. What should have served convenience then becomes a source of frustration. Inexperienced new colleagues or impatient customers in particular may suffer from this experience. How, then, do you create a useful and accurate helpdesk using AI? For our internal needs, Quatronic have developed a chatbot that finds its information using RAG, takes the conversation history into account and actually saves us time. This blog explains how.
Chatbots employ large language models (LLMs) to predict which answer to a given question is most likely the best based on public sources. If you want to search within a particular system, for instance amongst your company’s documents, this may result in falsehoods. After all, your organisation’s internal information will not be found on the internet, such that the LLM then ‘hallucinates’ wrong answers. To forfend this, we have used the technique of retrieval-augmented generation, or RAG.
The result is a chatbot that does not fabricate untruths when it cannot determine the answer to a question, but rather specifically makes use of the files and data provided. Moreover, it mentions the source utilised, so that you can verify whether or not the final answer is correct afterwards ― without having had to look it up beforehand. This is ideal for a new colleague asking standard questions with unequivocal answers, or for a customer-support service where a human employee would otherwise have to deal with the same complaints repeatedly.
The method underlying RAG
RAG is not an alternative to LLMs but concatenates several of them to enable targeted queries in a predetermined set of sources. The technique consists of several steps, each effected by an existing LLM.
1. Indexation
The sources that a chatbot may use, are subdivided into smaller parts, the so-called indexing process. This can be a single slide of a PowerPoint presentation or one page of a pdf file. Thus, the chatbot can find its information on a detailed level.
2. Meaning
Of course, the chatbot has to know what these fragments ‘mean’, both in and of themselves as well as in relation to each other. As such, mathematical quantities are attached to them that the chatbot can then read.
These numbers measure the semantic meaning of the data fragments (what meaning do they contain?) and determine their lexical ordering (which letters are used?). For the experts: each part is mapped to a vector of numbers within a ‘semantic vector space’ of high dimension.
3. Interpretation
Next, the chatbot processes the user’s question according to steps 1 and 2. Then, the ‘semantic distance’ between the question and source-material fragments is computed using aforementioned mathematical values. The question and source data are also compared lexically: to what extent do the words used agree? Based on these measures, a top 50 emerges comprising those sources that match the question’s meaning and vocabulary closely.
4. Re-ranking
Finally, a more powerful LLM compares this top 50 to the question once more. It reduces the top 50 to a top 12 that then leads to the chatbot’s written reply.
Reliability
Executing RAG is more complicated than invoking a single LLM, so the effort must be justified. Therefore, Quatronic’s chatbot first checks whether RAG would be useful in the first place. If the user only types ‘hello’, that is obviously not the case. Furthermore, it applies so-called contextualisation, meaning that the question is interpreted within the context of the preceding conversation, if there was one. Determiners and adverbs such as ‘it’, ‘here’ or ‘this’ would otherwise be incomprehensible to the LLM.
Yet how good are the answers obtained via RAG? For our chatbot, we first devised a list of questions and answers to test it. It answered these questions correctly in 80 to 90 percent of the cases. The primary bottleneck is the source data themselves: the clearer those are, the more accurately the chatbot can connect them to the question. Two different documents’ being quite similar (in language or content) may also lead the LLM astray. In that case, it does not know how to choose between the two and arrives at a vague if not altogether false answer. As such, it is important to state the source used together with the answer, so that the the user can verify the answer.
Especially for customer support, this reliability is crucial. For instance, in a questionnaire conducted by the Nationale Voice Monitor 2026, only 12 percent of participants indicated that chatbots provide good answers to their questions. The annoyance relating to chatbots is moreover increasing: half of the respondents experiences some. This is roughly the same percentage as that indicating frustration with waiting times on the telephone, which is lower than it used to be. (HP/De Tijd, October 2026, p. 58.) Therefore, it is evermore important to ascertain that the replies not only are factually correct but also fit within the context of the questions and the overall conversation.
Further developments
After its initial development phase, our chatbot was only able to read information from documents. The next step is to enable it to access relational databases. That is, data in the form of tables with relations between them. For this purpose, it will receive an AI agent (see our blog about AI agents) that can use SQL to read such data. Soon, an employee will not only be able to enquire how many days off we have per year, but also how many he has used up himself in the last six months.
The chatbot this becomes a practical tool for our colleagues. Whether they need specific or general information, they can obtain it easily and quickly. This leaves them more time to focus on their daily job.
Curious as to howyou can also employ such a chatbot? Give us a ring!
Get in touch with
Auke
