MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

AI 中文解读

**核心亮点：** 让大语言模型像“变形金刚”一样，无论设备内存大小都能自动调整精度，一次校准就能服务所有场景，效果还更好。 **通俗解读：** 大语言模型（比如ChatGPT的底层技术）通常很占内存，为了在不同设备上运行，需要“压缩”模型——就像把高清照片压成小图。但不同设备的内存大小不同，压缩得太狠会变模糊（降低准确率），不够狠又装不下。传统做法是针对每种设备单独调优，费时费力。MixQuant就像给模型装了个“智能变速器”：一次校准后，它能自动为每层神经网络分配合适的内存占用，无论设备是手机、电脑还是云端，都能在保证准确率的前提下运行。更厉害的是，它还考虑到层与层之间的相互影响，避免“眉毛胡子一把抓”的简单压缩。 **实际影响：** 这意味着未来你用的AI助手会更流畅：在手机上和语音聊天时反应更快，写文章时回答更准确，甚至老旧设备也能流畅运行最新模型。对企业来说，部署成本大幅降低，比如一家公司想开发AI客服，只需一次优化就能适配所有用户设备，开发周期从几周缩短到几天。对用户而言，AI服务的响应速度和稳定性会显著提升，覆盖更多生活场景，比如语音助手、智能翻译或教育辅导。

arXiv:2607.23047v1 Announce Type: new Abstract: Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget. In practice the budget varies across deployments and is unknown at calibration time. Adaptive quantization addresses this with one offline calibration that serves any budget, yet current methods score layer sensitivity in a manner that does not consider its dependency on quantization levels of other layers. We show that a layer's sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation. We propose MixQuant, a technique-agnostic adaptive framework that wraps any base quantizer. MixQuant marginalizes each layer's distortion over random quantized upstream configurations to obtain budget-agnostic scores, calibrates the quantizer's parameters on plans the allocator itself produces, and penalizes allocations that leave layers at the lowest bitwidths. A single greedy pass then serves any budget at deployment. Across Llama-3.2-3B, Llama-2-7B, and Mistral-7B under AWQ and GPTQ, MixQuant outperforms adaptive and mixed-precision baselines in every setting, improving average accuracy by up to 8 points and reducing perplexity from 12.43 to 10.70 at the tightest budget, while matching an ILP solver at negligible deployment cost.