Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes

Gravellier, Zhang, Wu et al. released a preprint study developing the open-source benchmark SeeFood, using 80 real meal photos from the SNAPMe dataset to evaluate 8 commercial and open-source multimodal large models, and re-evaluating three shortlisted models on the full dataset of 1,680 photos and 275 person-days. The paper reports that GPT-4o mini had the lowest error, with a significant advantage on carbohydrates; however, the mean absolute percentage error still ranged from 52% to 90%, with the direction of bias varying by model. The mean absolute error for carbohydrates was 13.3 g (95% confidence interval 12.4 to 14.3), and only 57% of meals fell within plus or minus 10 g, the clinically relevant threshold for insulin dosing in type 1 diabetes. Calorie error for lunch and dinner was about 55%, comparable to human estimates; only beverage images, due to dissolved sugar, had errors as high as 312%. The authors state that current models can extract meaningful dietary information, but errors in nutrition and portion estimation limit their use for precision diabetes management.