Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

[HF Papers] Shuyuan Tu, Qi Tian, Yinming Huang and others propose Prism, a dynamic sparse attention framework for training 2K native joint video-audio generation models. The paper states that the computational cost of full attention grows quadratically with resolution, and that attention dispersed onto redundant tokens dilutes the learning signal and damages pretrained priors; existing sparse attention methods either target training-free acceleration or overlook the structural characteristic that cross-modal interaction is concentrated in sounding regions. Prism organizes the token sequence into spatiotemporal macro-regions, estimates local information structure based on video feature channel variance and the feature norm of audio-to-video cross-attention, dynamically allocates block shapes for each region, and employs a hybrid block selection strategy to determine per-query sparsity. Experiments show that Prism improves training speed by 2.5x over full attention, and its generation quality also surpasses the latter.