Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification

AI 中文解读

这篇论文给强化学习装上了一个“双引擎”系统，最吸引人的地方在于：它让AI的“决策层”和“执行层”能同时学习，而不是像以前那样一层层慢慢来，学习速度大幅提升。通俗来说，以前的AI学做一件事，就像新人先学“大目标”再学“小动作”，两个阶段互相等待，耗时又费力。如今这个新算法让“指挥官”和“执行者”同步摸索，甚至利用“执行者”多步行动后的结果来辅助“指挥官”调整策略。研究还发现，只要底层运动规律满足两个特定条件，这套并行学习法就能严格证明比非分层方法更省数据，尤其在奖励稀疏、目标明确的任务里优势显著。这意味着未来训练机器人收拾房间、自动驾驶在陌生路况导航这类复杂任务时，AI不再需要海量试错就能快速上手，玩家在游戏中遇到的AI队友也会变得更聪明、反应更自然，而普通用户用到的智能助手、智能家居，响应速度和决策质量都可能迎来肉眼可见的提升。

We present HBPI-UCRL, a model-based algorithm for hierarchical reinforcement learning (HRL) that learns high-level and low-level policies in parallel. HBPI-UCRL exploits the fact that a high-level transition corresponds to a multi-step transition at the low level. We introduce two conditions on the low-level dynamics that are sufficient to make parallel HRL learnable. When these conditions hold, we prove that HBPI-UCRL has a polynomial sample complexity in the problem parameters. In the sparse-reward, goal-directed setting, our sample complexity upper bound for HBPI-UCRL is strictly lower than that of its non-hierarchical counterpart, providing theoretical justification for the empirical success of HRL.