HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

AI 中文解读

HOBA这项研究给在线广告竞价系统装上了“智能大脑”——以前广告竞价全靠多个离线训练的专家模型，不仅调参费时费力，还无法应对市场变化。现在，HOBA用分层强化学习框架，把策略推理、模型选择和执行分成三个时间层级：高层用大语言模型像指挥官一样根据历史经验调整参数，中层用智能算法动态挑选最佳专家模型，底层再由PID、决策转换器等专家执行竞价。这种设计把在线学习限定在“选哪个专家”而不是“怎么出价”，既降低了试错风险，又保持了灵活适应能力。实际测试中，HOBA在大型线上部署中让目标成本提升了3.6%，意味着广告主能花更少的钱获得更好的投放效果。对普通人来说，未来你看到的广告会更精准、更合心意，而广告平台也能减少人力调参的麻烦，整个广告生态变得更聪明、更高效。

arXiv:2607.24779v1 Announce Type: new Abstract: Online advertising bidding systems typically deploy multiple offline-trained expert models (e.g., PID controllers, model predictive control, offline RL policies) but face two critical limitations: lack of online adaptability to non-stationary auction markets, and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints. We propose HOBA (Hierarchical On-policy Bidding Agents), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales. At the high level, a large language model infers hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop with historical experience retrieval. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias. At the low level, a dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability. Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines. In a large-scale online deployment, HOBA delivered substantial business value, achieving a +3.6\% increase in target cost, proving the effectiveness of our hierarchical multi-agent bidding paradigm.