Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study

Rui, Ma, Stepanov et al. published a retrospective cohort study evaluating the reliability of a multi-large language model (LLM) consensus workflow for automatically determining outcome labels from radiology reports. The study included 6,718 brain MRI reports from 5,856 patients; the expert-adjudicated reference set contained 680 reports, of which 372 were normal, 90 had white matter hyperintensity only, and 218 were abnormal. The paper states that when Qwen3, Kimi-Linear, and GPT-5 (or DeepSeek-R1 in the open-source combination) were all required to agree before assigning a normal or abnormal label, with all other cases referred for manual review, both ensembles achieved 100% precision in automatic classification on the reference set; the strongest single model, GPT-5, achieved normal and abnormal precision of 98.9% and 94.1%, respectively, with 17 cases discordant with expert labels, whereas the ensembles had none. In the full cohort, among reports classified by the proprietary-model ensemble as normal, indeterminate, and abnormal, 1.6%, 29.7%, and 51.3%, respectively, were accompanied by central nervous system-related ICD-10 diagnoses; the corresponding figures for the open-source ensemble were 1.6%, 31.1%, and 51.6%. The authors state that this strategy can reduce the burden of manual adjudication.