cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

AI 中文解读

1. 核心亮点：科学家找到了一种新方法，让多个AI模型像“专家团队”一样高效协作，在不大幅增加算力成本的前提下，显著提升语言模型的理解和生成能力。 2. 通俗解读：过去，训练一个超大的AI就像让一个人记住全世界所有的知识——模型越大，计算量就越大，成本也直线上升。这篇研究提出的“cMoLLM”技术，相当于把AI拆分成一群各有所长的“小专家”，每个专家只负责自己擅长的领域。当用户提问时，系统会像“智能导演”一样，自动挑选最合适的几位专家共同解答，并让它们像拼乐高一样无缝配合。与以往方法相比，这种新方式让专家之间的“沟通”更顺畅、训练更稳定，而且计算资源分配更合理，避免了重复劳动。 3. 实际影响：这意味着未来我们使用的AI助手——无论是写邮件、查资料还是做翻译——可能会更快、更聪明，同时背后的公司（比如开发AI的科技企业）也不必无休止地堆硬件。对普通用户来说，最直接的感受可能是：小公司也能负担得起强大的AI服务，而大模型的回复将更准确、更少“犯糊涂”，甚至手机上的AI应用也能跑得更流畅。长远看，这项技术有望降低AI的使用门槛，让更多人享受到智能化的便利。

arXiv:2607.22577v1 Announce Type: new Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes. We aim to scale capacity through MoE-style mixture throughout the LLM pipeline rather than only the FFN. Prior pipeline-level approaches include ParaScale, which introduces virtual tokens and parallel streams but incurs substantial overhead and suffers from homogenized routing and gradient collapse, and AltUp, which uses an auxiliary prediction branch but offers limited adaptivity and slow convergence. We establish that MoE-style mixture layers can be reformulated as variable-kernel dynamic convolutions, where each expert corresponds to a $1{\times}1$ convolutional kernel and routing implements input-conditioned kernel aggregation. Building on this equivalence, we introduce cMoLLM: a convolutionally gated mixture-of-LLMs that routes over end-to-end streams through fully differentiable dynamic convolution. In GPT-2-style models trained on FineWeb, cMoLLM improves language modeling perplexity and downstream GLUE and SQuAD accuracy under matched compute, with better stream utilization, more stable optimization, and favorable scaling compared to ParaScale- and AltUp-style baselines.