Does Every Instance Count Equally? Order Effects in LLM Multi-Instance Processing

[arXiv] Jingxuan Chen, Mohammad Taher Pilehvar and Jose Camacho-Collados have published a paper stating that users often ask large models to summarize multiple independent inputs and give a single aggregated answer, i.e., multi-instance processing (MIP), and the correct result should be independent of instance order. The authors evaluated 15 large models on five datasets under seven ordering patterns, with each instance set fixed across patterns, and found significant order sensitivity: accuracy can differ by about 90 percentage points between the best and worst ordering patterns; when most labeled instances are placed earlier, accuracy is generally higher, and when they are placed in the middle or later, it is worse. The reasoning traces of some models and gradient-based attribution show uneven positional weighting: the traces mention earlier instances more often, and attribution scores are usually higher at the beginning and end than in the middle. The authors state that instance order can systematically bias aggregated predictions, and that evaluating MIP needs to cover different input permutations.