Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

[arXiv] Ming Zhong, Stacy Li, Nicholas Carlini and Matthew Jagielski published a paper stating that they introduce Agent4Sci, a long-horizon test platform that measures whether frontier agents can, under in-the-wild conditions, improve on the published best models on scientific machine learning benchmarks, covering a total of 20 benchmarks across 11 domains. The authors state that today's frontier agents advanced the published best on every benchmark, with Claude Opus achieving a mean relative improvement of 10.4%, and no run was flagged for academic misconduct; however, these gains came almost entirely from machine learning engineering tricks rather than research-level insight. Trajectory analysis summarizes limitations including novelty calibration bias, insufficient resource utilization, and rapid collapse into local engineering sweeps. The authors state that lightweight plug-in skills such as research ideation, experiment management and timed checklists raised Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, and brought more diverse and more domain-relevant interventions.