WAKMan picks KV-cache eviction policy from live traffic, lifts hit rate 5 points over LRU

[arXiv] A preprint introduces WAKMan, a workload-aware KV cache manager. Authors Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese and Wesley Harris said prefix caching lets LLM servers reuse KV states to speed prefill for requests sharing a prefix, but with limited memory the eviction policy decides the benefit. Across 20 traffic mixes, the authors said no single policy is always best: vLLM's default LRU delivers 12% higher cache hits than the frequency-oriented S3FIFO on agentic-heavy mixes but trails it by up to 22% elsewhere, and a single workload parameter can flip the winner. WAKMan selects an eviction policy from live traffic without knowing request types or reading prompts; according to the authors, it improves hit rate over LRU by 5 percentage points on average and has the lowest mean and worst-case optimality gap of all policies. The work is a preprint and has not been peer-reviewed.