Decision-only model judged medical AI answers at a fraction of frontier cost, study says

[medRxiv] A decision-only model that returns structured probabilities rather than text can serve as a low-cost judge of medical AI answers, according to an OSF pre-registered evaluation of TypeSafe Jev 1.13. The study tested the model on tasks from five validated clinical benchmarks: HealthBench, When Rubrics Fail, MedVAL, CounselBench and Real-POCQi, covering 25,091 units and 213,913 model calls. According to the authors, Jev's agreement with expert labels averaged 64.7% over six tasks and 66.1% on 3,614 matched units, versus 65.0% for Opus and 60.7% for Sol, and it was non-inferior to both on four of six tasks. Jev judged the whole pool in 17 minutes for US$1.46, at 1/34 of Sol's per-judgment price and 1/152 of Opus's. Omissions remained hard: at the same false-alarm rate, Jev missed 55.6% of planted omissions and Opus 48.1%. The preprint has not been peer-reviewed.