PIVOT Explained — From Paper to Working Code in 10 Minutes

PIVOT技术给AI长文本处理带来一个巧妙突破：不用修改模型参数，只用一个分组扫描的小技巧，就能让处理速度提升4倍，延迟降低近40%。核心亮点在于，它解决了AI在处理超长文本时“假稀疏”的尴尬——以往号称高效的方法，实际还得逐个检查所有字词，效率根本没提升。通俗来说，AI在处理长篇内容时，需要从大量文字中找出关键信息。传统做法每个问题都要扫描全部文本，工作量巨大。PIVOT发现，相邻的问题其实关注的内容高度重叠，于是它把这些问题打包成组，只派一个代表去扫描全文，再把结果共享给组内成员。这样一来，扫描次数大大减少，速度自然快了起来。这项技术对普通用户意味着实实在在的体验提升。比如你用AI助手分析百万字的书籍或论文，等待反馈的时间可能从几分钟缩短到几十秒；智能客服在回答长对话时，也能更快理解上下文。由于PIVOT无需重新训练模型，任何现有的AI服务都能直接应用，未来我们可能随时享受到更流畅的聊天、更快速的文档处理和更智能的搜索体验。

You enabled sparse attention. Your model still chokes at 128K tokens. The indexer is why — and PIVOT fixes it without touching your weights. <h2> TL;DR </h2> <ul> <li>Sparse attention's indexer scores all L tokens per query → still O(L²)</li> <li>PIVOT groups nearby queries (which select ~90% overlapping top-k tokens), runs one proxy scan per group → O(L²/g)</li> <li>Result: 4× indexer speedup, 1.6× end-to-end latency reduction on DeepSeek-V3.2 and GLM-5.1</li> <li>Training-free: plug into existing DSA models at inference time</li> <li>Two modes: PIVOT-Reuse (max speed) and PIVOT-Refine (matches dense indexer accuracy)</li> </ul> <h2> The Problem </h2> Dynamic Sparse Attention (DSA) should make long-context inference fast. Score all tokens → pick top-k → attend only to those k. Complexity drops from O(L²) to O(L·k). Except scoring all tokens is itself O(L²). The "indexer" does a full O(L) scan per query position. With L queries, you're back to O(L²). At 100K tokens, the indexer dominates latency. Sparse attention becomes a lie. <h2> How It Works </h2> Observation 1: Adjacent queries share ~90% of their top-k token selections — they process nearly identical context. Observation 2: Indexer scores are long-tailed — a proxy query produces a reliable candidate set. PIVOT's algorithm: <div class="highlight js-code-highlight"> <pre class="highlight python"><code>group = [q_i, q_{i+1}, ..., q_{i+g-1}] proxy_q = mean(group) # ONE scan instead of g scans scores = proxy_q · K[:i] # O(L) C = top-K(scores) # candidate set, K = 2 × top_k # per-query refine (PIVOT-Refine) for q in group: refine_scores = q · K[C] # O(K), not O(L) final_indices[q] = top-k(refine_scores) </code></pre> </div> Indexer cost: O(L²) → O(L²/g). With g=8 that's 8× fewer full scans. <h2> Show Me The Code </h2> <div class="highlight js-code-highlight"> <pre class="highlight python"><code>import torch import torch.nn.functional <span c