The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Yu Lin, Yiming Wang, Runyuan Cai and others propose the streaming MoE inference engine Edge0, and the paper claims it can run a 35B-class MoE model on a single 24GB machine at 20tok/s, with peak active memory kept within 3GiB. The authors point out that a 35B-class model has 19.5GB of weights after 4-bit quantization; sparsity only reduces the amount of computation per token, not the bytes that must reside in memory; and naive SSD offloading does not work, because the experts for layer N+1 must be selected before the output of layer N is produced, so the reads cannot be hidden behind computation in advance. Edge0 uses a pre-routing head to predict the next layer's routing one token ahead, and uses the prediction directly as the routing, so that the prefetched expert set equals the actual routed set and no experts are dropped; it also uses an unmerged recovery LoRA trained on the student path to compensate for the quality loss caused by int4 quantization and routing substitution. The paper claims that on five public benchmarks it averages within a few percentage points of its fp16 teacher model, that the 8B tier also runs in the same framework, and that the framework, checkpoints and adapters have been open-sourced.