HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

AI 中文解读

核心亮点：科学家找到了一种让AI大模型在极低精度（4位）下训练时，性能几乎不输高精度（16位）的方法，且计算成本大幅降低。通俗解读：想象你用手机拍夜景，为了省内存把照片压缩得很小，结果暗部细节全丢了——这就是AI在“节省算力”时遇到的尴尬。通常AI训练需要高精度计算（类似高清画质），但费电又慢。这次研究者发现，问题不在训练本身，而在于AI“生成练习题”（即 rollout）时的数字记录方式：4位精度的“刻度”太粗，导致很多小数值被“四舍五入”成了零。他们发明了一个叫“残差修正”的小技巧，就像在压缩照片上加一个“暗部补偿滤镜”，只花极少的额外计算，就能把丢失的细节捞回来。加上他们设计的HiFloat4格式，用三级分级放大，让4位精度也能细腻地表示不同量级的数据。最终，AI用超低精度训练后的表现，和用高精度的差距从5%缩小到了1%。实际影响：这意味着未来运行大模型（比如智能客服、AI写作助手、自动驾驶决策）的硬件成本可能大幅下降——就像普通手机也能流畅运行以前需要旗舰机才能跑的AI应用。普通用户会体验到更便宜的AI服务，开发者也能在手机或物联网设备上部署更聪明、反应更快的AI。一句话：省电省内存，但AI依然很聪明。

arXiv:2607.26515v1 Announce Type: new Abstract: We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.