Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training

AI 中文解读

1. 核心亮点：这项研究让AI在算力不足的情况下，也能高效“修炼”出更强的工具使用能力，性能直逼顶级训练方法。 2. 通俗解读：训练AI助手就像教学生做题，传统方法要一步步“反向算答案”，特别费电脑内存，就像用一杯水去浇灌整片森林。新方法CoPES干脆把AI的“大脑”拆成很多小块，让每块独立学习、互相配合，就像团队分工协作，既省水又能把整片森林浇透。实验显示，在同样有限的硬件条件下，这个方法能恢复顶级训练方案九成多的效果，而内存占用还不到它的八分之一，相当于用小型跑步机练出了接近专业运动员的成绩。 3. 实际影响：以后在普通电脑或小公司里，也能训练出会算数学题、能查资料、能对话的智能助手了。这意味着AI服务成本会更低，我们用的手机助手、线上客服可能会变得更聪明，响应更快，而厂商也不用拼命堆服务器，最终受益的还是普通用户。

Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive. Evolution strategies (ES) enable memory-efficient full-parameter post-training without backpropagation and can eventually match the performance of gradient-based reinforcement learning (RL). However, resource-constrained settings typically offer only a few GPUs, so the high GPU-hour requirements of ES translate into prohibitively long training times. To address this, we introduce Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that decomposes the full parameter space into lower-dimensional subspaces and searches over them cooperatively to improve optimization efficiency. We post-train a Qwen3.5-4B tool-using agent for the math task and evaluate it on five benchmarks of varying difficulty. Under the GPU-hour budget of full-parameter GRPO's best validation checkpoint, CoPES recovers 92% of GRPO's validation-accuracy gain, versus 67% for standard ES, while its theoretical GPU memory requirement is less than one-eighth that of full-parameter GRPO. It consistently outperforms standard ES and LoRA-based GRPO on all evaluated pass@k metrics across the five benchmarks. Additional experiments further show the advantage of CoPES on the question-answering task. These results demonstrate an improved trade-off between memory requirements and training time for agentic LLM post-training under resource constraints. The code is open-sourced in https://github.com/MetaronWang/CoPES