Cost Accounting for Reactive Computational Graphs: Exhaustive Sweeps, Sequential Mutation, and the Backward-Locality Gap

AI 中文解读

**核心亮点：** 这项研究首次精确算清了神经网络“改一个地方要重算多少”的成本账，颠覆了此前认为效率翻倍的简单直觉。 **通俗解读：** 想象你有一台精密的机器，每当你调整其中一个小螺丝，整台机器都要重新测试一遍效果。这就是做AI模型调试时“激活修补”和“消融研究”的现状。论文发现，这种“改动-重算”的效率并不是固定的：如果改动靠近输出端，速度提升只有1.8倍左右，远低于理想中的2倍；而改动靠近输入端时效率更差。更关键的是，在反向传播（训练模型常用的方式）中，这种加速效应几乎完全消失，效率反而比独立重算还低。研究团队用Julia语言实现的引擎NeuroDSL验证了所有理论推导，误差为零。 **实际影响：** 这项技术短期内主要影响AI研究者和工程师——他们做模型调试、找网络“电路”、做消融实验时，可以更精确地预估计算成本，避免盲目相信“并行优化”的效果。长远看，更精准的成本核算有助于降低AI模型的研发功耗和算力开销，让企业用更少的GPU完成同样的分析工作，最终可能促使更高效的AI训练框架出现。对普通人来说，这意味着未来AI应用背后的研发成本有望下降，但日常使用体验不会立刻改变。

arXiv:2607.18323v1 Announce Type: new Abstract: Exhaustive site-by-site interventions on a neural network's computational graph -- activation-patching sweeps, circuit-discovery searches, systematic ablation studies -- mutate the graph at every candidate site, and their cost is dominated by recomputation after each mutation. On a reactive graph engine whose invalidation provably touches exactly the downstream cone of a mutated node, we give a complete cost accounting for such workloads. First, the aggregate speedup of an exhaustive sweep over independent full recomputations is not a universal constant: if per-layer weight varies regularly with depth at Karamata index q, the ratio converges to (q+2)/(q+1) when weight concentrates near the output and to q+2 near the input, recovering 2 only in the depth-uniform case; a wall-clock corollary predicts a ceiling of about 1.79, below 2, until interpreter overhead is compiled away. Second, we prove the exact cost of a sequence of persistent mutations, never undone between insertions: the interleaved cost exceeds the isolated sum by an exact overcount summed over comparable site pairs, with closed-form extremes over insertion orders, while batched application is order-independent and sub-additive, costing exactly the union of the sites' cones plus the fresh nodes. Third, we prove the exact mirror of forward locality for the backward pass, showing it collapses the aggregate speedup to 1 under backpropagation on architectures without long skip connections. Every identity is validated on NeuroDSL, a reactive graph engine in Julia: measured sweep ratios converge to the predicted limits under four cost profiles; the training-mode ratio collapses to 1 at the predicted rate; and all 18 per-graft sequential costs and the batched total match the closed forms at zero tolerance across three insertion orders.