From Prompts to Posteriors: Can LLM Agents Perform Bayesian Inference in Context?

[arXiv] Linus Filbry, Luckeciano C. Melo and Yarin Gal have published a paper stating that existing work often assumes large models cannot perform principled Bayesian belief updating in context. The authors construct an evaluation framework covering qualitative epistemological examples, quantitative benchmarks for Bayesian inference, and an analysis of the impact of non-Bayesian updating on downstream tasks, to test the models' internal consistency and their ability to recover externally given posteriors. The authors state that across scenarios large models generally violate Bayesian rules when updating beliefs in context; however, after being equipped with modern reasoning capabilities and code execution, performance improves substantially, and can even approach posteriors that are analytically intractable. The paper states that current large models, once given appropriate tools, can already be used for complex belief updating in scientific discovery.