Genetic Perturbation Prediction Models Fail Out-of-Distribution: A Rigorous Benchmark and Systematic Evaluation

Tiwari, Mahajan, Bellad and others have released a preprint study proposing a benchmarking framework called EL-PERTURB, which stress-tests 20 perturbation prediction models, covering six classes of architectures, and evaluates them on essential-gene and genome-wide Perturb-seq screening data. The authors state that existing benchmarks mostly rely on random train-test splits, which mask the systematic failure modes of models under real out-of-distribution (OOD) conditions; EL-PERTURB adopts a stratified OOD splitting strategy that places the training set and the test set in functionally distinct regions of transcriptomic space. The study reports that model performance drops substantially under OOD conditions, and that architectural complexity—including single-cell foundation model embeddings—offers no reliable advantage over well-tuned simple baselines. The authors state that the framework provides a rigorous benchmark covering different levels of generalization difficulty for perturbation-effect prediction, and can guide the construction of models that can truly replace wet-lab screening.