EM-BENCH benchmark finds no LLM fully guideline-concordant; top score 98.6%

[medRxiv] A medRxiv preprint reports the development of EM-BENCH, a physician-validated, deterministically scored benchmark for assessing whether large language models reliably apply American College of Emergency Physicians (ACEP) guidelines to clinical decisions. The benchmark comprises 210 binary clinical vignettes covering all 70 critical questions across 25 ACEP clinical policies, and the study evaluated 149 LLM configurations representing 79 base models. According to the authors, no configuration achieved complete guideline concordance; the highest-performing configuration answered 207 of 210 vignettes correctly (98.6%). Closed-access models outperformed open-weight models (median accuracy 91% vs 63%; Hodges-Lehmann location shift 26 percentage points; 99% CI 20 to 33; adjusted P<0.001). Among open-weight model families, a 10-fold increase in parameter count was associated with a 14-percentage-point increase in accuracy (99% CI 7.7 to 20; adjusted P<0.001), and accuracy rose 8.9 percentage points per release year across all models (99% CI 3.6 to 14; adjusted P<0.001). Neither enabling a thinking mode nor higher reasoning effort consistently improved accuracy. The authors said contemporary LLMs showed improving but incomplete ability to apply ACEP guidelines.