OA-RAG agent combines ontology formal reasoning with LLM reasoning for faithful answers

[arXiv] A preprint by Hui Yang, Jiaoyan Chen and Yongsheng Gao proposes OA-RAG, an agent that integrates formal reasoning with LLM reasoning by drawing on both the logical structure and informal annotations of domain ontologies. According to the paper, OA-RAG reports for each answer the ontology observations used as evidence and a model-assessed grounding level (entailed, supported, related or unrelated) that characterizes the strength of that evidence. The authors said that across benchmarks derived from three large-scale ontologies and six real-world medical question-answering tasks, OA-RAG achieved the best overall performance among the evaluated ontology-fine-tuned and popular RAG methods, with particularly strong results on reasoning-intensive tasks. The paper also said the reported grounding levels offer a useful indication of answer reliability, with stronger grounding generally associated with higher accuracy, and that the results demonstrate the value of ontologies as knowledge sources, with OA-RAG competitive with and sometimes outperforming domain-specific RAG systems that draw on sources such as textbooks. The work is a preprint and has not been peer-reviewed.