Richer database context boosts LLM-generated EHR phenotyping SQL, study finds

[medRxiv] A preprint study evaluated large language model generation of type 2 diabetes phenotyping SQL and the effect of database implementation context. According to the authors, eight publicly accessible LLM-based systems generated SQL under progressively richer prompts using synthetic and University of North Carolina (UNC) Health electronic health record data, both based on the OMOP model, and outputs were compared with the eMERGE T2DM reference phenotype on SQL executability, F1, positive predictive value and recall. Fewer than 55% of limited-context synthetic outputs and no limited-context UNC EHR outputs executed without modification, the authors said. Detailed codebooks (L3) raised executability to 81.3% in synthetic OMOP and 100% in UNC EHR, while adding sample rows (L4) raised synthetic executability to 100%. Limited-context recall was as low as 0.01, whereas high-context outputs achieved recall of at least 0.99 at synthetic L4 and above 0.90 at UNC L3, although UNC PPV remained substantially lower than recall. At the highest available context levels, Beta F1 ranged from 0.72 to 0.99 in synthetic OMOP and 0.57 to 0.68 in UNC EHR; all UNC models exceeded the diagnosis-code baseline (F1=0.54). The top synthetic performer fell from F1=0.99 to 0.64 on UNC EHR. The authors concluded that richer database context was associated with more consistent SQL executability and recall, while model rankings varied across datasets, and that limited PPV in real-world data underscores the need for local validation.