MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

AI 中文解读

多模态大模型看视频或图片时，需要处理海量的视觉信息，这会让内存很快被“塞满”，就像手机同时打开几十个网页导致卡顿。现有技术试图在模型“预读”阶段就判断哪些信息重要、哪些可以丢掉，但研究发现，这些预读的判断在模型真正“思考回答”时常常失效，导致筛掉了关键的视觉细节，比如把图片里识别路标、物体形状的重要像素给丢了。对此，研究者提出了MM-ShiftKV，这套方法无需额外训练，却能巧妙地在预读阶段“预判”模型后续思考时可能关注的信息范围，从而更精准地保留真正关键的视觉内容，在极低内存预算下依然不丢失理解力和推理准确性。这项技术一旦落地，普通人使用AI分析图表、看医学影像、甚至用手机扫菜谱做菜时，响应会更快，且不会因为AI“断章取义”而给出离谱答案。对开发者和企业来说，意味着能在大幅降低服务器硬件成本的同时，让AI应用保持高质量体验。

arXiv:2607.22586v1 Announce Type: new Abstract: Key-Value (KV) caching is essential for efficient inference in multimodal large language models (MLLMs), yet its memory footprint grows linearly with context length and becomes a major bottleneck due to the large number of visual tokens. Recent prefill-stage KV selection methods estimate KV importance from prefilling statistics, implicitly assuming that prefilling-time queries are representative of those encountered during decoding. We show that this assumption breaks down in multimodal inference, where decoding-time queries exhibit substantially larger variance than prefilling-stage representations, leading to unstable KV importance estimation under tight cache budgets. As a result, small ranking errors can disproportionately discard semantically critical visual tokens and degrade grounding and reasoning performance. We propose MM-ShiftKV, a training-free, decode-aware and strictly prefill-only KV selection method. MM-ShiftKV approximates decoding-time query behavior during prefilling by constructing variance-expanded query proxies and estimates prompt KV importance based on their aggregated attention mass. Experiments on multimodal benchmarks demonstrate that MM-ShiftKV consistently outperforms existing methods under strict KV-cache budgets. Our code is available at https://github.com/zjuDBxAI/MM-ShiftKV.