Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation

[arXiv] Karim Abdel Sadek, Federico Di Gennaro and Fanny Yang state in a paper published on arXiv that in recursive self-distillation, where the model obtained from each round of self-distillation serves as the teacher model for the next round, a safe model—even if trained only on safe content it generates itself and with harmless prompts—will become increasingly misaligned on various harmful requests after multiple rounds. The authors call this "progressive misalignment" and measured the phenomenon on the Qwen3 and Phi-4 model families using SORRY-Bench and StrongREJECT. Replacing fixed prompts with simulated user feedback with benign intent also produces this problem. The authors also used gradient-based methods to find benign prompts that can trigger the phenomenon, indicating that harmful training prompts are not required.