Study proposes directional metrics for AI-radiologist disagreement, net non-agreement ranging from -4.18% to +10.74%

[medRxiv] A preprint study proposed directional metrics — Positive Non-Agreement (PNA, AI correct and radiologist incorrect), Negative Non-Agreement (NNA, radiologist correct and AI incorrect) and Net Non-Agreement (NENA, their difference) — to quantify disagreement between AI platforms and radiologists against a common reference standard. According to authors El Kaddouri et al., applying the method to three previously published AI-radiologist studies on pulmonary embolism and mammography yielded NENA values ranging from -4.18% to +10.74%, decreasing with radiologist experience: +6.27% for junior residents and +1.45% for consultants, a gradient driven primarily by declining PNA with a modest increase in NNA. The authors also defined a reclassification fragility index measuring how many case reclassifications would remove statistical support for the direction of NENA; it was defined for six of seven comparisons, ranging from 0.34% to 4.31%, but not for the weakest comparison, whose confidence interval for NENA included zero. The findings have not been peer reviewed.