Do AI-for-Science Agents Preserve Model Safety?

[arXiv] Michael Yew Chuan Lee, Manya Sriram, and N. Benjamin Erichson et al. argue that when large language models are connected to agent frameworks geared toward the scientific domain, the models' original safety behavior is difficult to maintain. The team evaluated five agent frameworks on 50 harmful chemistry requests and tested direct prompting, two simple adversarial prompts, and one adaptive attack, covering four recent GPT variants. The results show that even under direct prompting, agent frameworks may raise the compliance rate for harmful requests; more critically, an adaptive attack that is largely ineffective against base models saw its success rate surge once a framework was introduced. The authors therefore conclude that the safety behavior of base models does not persist after being connected to a framework, and that as the capabilities and autonomy of AI-for-Science agents increase, the community needs to use domain-specific safety benchmarks to evaluate and align agents.