vLLM authors claim PagedAttention lifts LLM serving throughput 2-4x

[HF Papers] A preprint dated Sept 12, 2023 by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng and Cody Hao Yu says high-throughput serving of large language models requires batching enough requests at once, but the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically; inefficient management wastes memory through fragmentation and redundant duplication, limiting batch size. The authors propose PagedAttention, an attention algorithm inspired by classical virtual memory and paging techniques in operating systems, and build vLLM, an LLM serving system they say achieves near-zero waste in KV cache memory and flexible sharing of KV cache within and across requests to further cut memory use. According to the authors' evaluations, vLLM improves throughput of popular LLMs by 2-4x at the same latency compared with state-of-the-art systems such as FasterTransformer and Orca, with the gain more pronounced for longer sequences, larger models and more complex decoding algorithms. The source code is publicly available on GitHub. The findings are from a preprint and have not been peer-reviewed.