MAREval multi-agent framework evaluates recommendation explanations; paper reports A/B gains in engagement and revenue

[arXiv] A preprint by Reza Yousefi Maragheh and co-authors proposes MAREval, a structured multi-agent framework that uses large language models to evaluate natural-language explanations in recommender systems. According to the paper, human evaluation is accurate but resource-intensive and impractical at the scale of modern recommendation platforms, while automated single-agent LLM methods suffer from prompt sensitivity and inconsistent outputs. MAREval orchestrates a planner agent using a new Chain of Debate prompting strategy to coordinate agent roles and enforce logically consistent evaluation plans, a moderator agent that mitigates prompt drift, an arbitrator agent that aggregates outputs from multiple evaluation rounds, and Monte Carlo sampling for robustness. The authors report that on the public TopicalChat dataset and a proprietary recommendation dataset, MAREval aligns better with human judgments than strong single- and multi-agent baselines, with substantially lower variability across repeated trials. They also report that in a large human-annotation gate MAREval met production quality thresholds where prior evaluators fell short, and that online A/B testing showed statistically significant improvements in engagement and revenue metrics. The findings are from a preprint and have not been peer reviewed.