Optimising Large Language Models for Medical Question-Answering: Reducing Hallucinations Using Retrieval Augmented Generation

Cameron Wishart, Valentin Peron, Gagan Biswas, Antoine Esman, Matthias von Rakowski, Arne Jacobs and Adrien Jacquot
Heriot-Watt University

You can try the chatbot at m-rag.gaganbiswas.com.

Abstract

Large language models can produce both strong and accurate answers in almost any subject. However, they often generate incorrect or factually unfounded responses. These inaccurate generations are called hallucinations, and they pose a critical risk when implemented in sensitive domains such as healthcare. This paper presents a medical chatbot, designed to ground responses of the large language model with the content of the NHS UK website. The system utilises a configuration-driven Retrieval-Augmented Generation (RAG) pipeline. This architecture incorporates a late adaptive chunking strategy, semantic retrieval via a vector database and a cross-encoder reranker. To ensure medical safety and factual grounding, the pipeline integrates an intent classifier, an LLM-as-a-Judge to verify response content and additional guardrails that evaluate reliability before displaying the response. Responses outside the scope of NHS knowledge are handled with a refusal mechanism. Evaluation is conducted through automated testing using LLM-as-a-Judge and human analysis of responses. The results demonstrated high faithfulness with 92% accuracy to the provided context, strong refusal and conversational capabilities.

Diagrams

Results

Mean scores on the generated dataset (built from the NHS UK website) and the HealthFC dataset. "Answered" excludes questions the system refused.

Conclusion

In conclusion, this paper describes a system that mitigates hallucinations of LLMs by enhancing the context of a question with the documents provided by a knowledge database. Regarding the database, the system will refuse to answer a question which is out of scope or ask for clarification if too vague. State of the art methods were used to optimise the pipeline, including query rewrites, document reranking and a context judge. This allows the LLM to receive enhanced context with the best chunks of from the database regarding the query. Then, guardrails and filtering are in place after the retrieval step to determine whether the model should proceed with generating the response, decline to answer the query, or request clarification in case the retrieved context is insufficient.

The results demonstrate a robust system that can reliably answer users' queries and generate appropriate citations or redirections for the user to follow. It is also capable of holding a conversation, referring back to previous conditions and contexts discussed.

The datasets for evaluation were challenging to curate as the NHS website has very particular and general information, meaning niche and obscure questions could be mistaken for out of context, as discussed in Section 7. This could be mitigated by incorporating additional data sources into the vector database to provide a deeper and more comprehensive set of trusted knowledge to address questions that the current system cannot answer.

Computational power was also a limitation, as the rag pipeline required multiple steps, and evaluation required substantial context to complete. The addition of a multi-step conversation evaluation dataset would provide further quantitative analysis of the system.

Another limitation was the time. The authors had to reduce their initial ambition due to the project's deadline. Features were explored, such as GraphRAG (Edge et al., 2025), LightRAG (Guo et al., 2025) and long term conversational history (Maharana et al., 2024; Wang et al., 2025), but couldn't be implemented due to the time constraints and are proposed as future work.

In this paper, the implementation of the conversational pipeline is both trustworthy and robust and can accurately provide NHS approved information to users seeking treatments or information about their conditions.