From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence

AI 中文解读

人类视频变机器人教材，这个叫Pegasus的新框架让机器人不用重新采集数据，直接“看懂”网上海量的人类操作视频就能学习技能。过去机器人和人类身体结构不一样，看着视频也学不会，就像让章鱼模仿人类弹钢琴。Pegasus的巧妙之处在于它当起了“翻译官”：把视频中人类的动作拆解成关键步骤，比如“拿杯子、倒水、端起来”，再转换成机器人能理解的操作指令。它还会用物理规则把关，跳过那些违反关节活动范围或碰撞的无效动作。这项技术意味着，未来训练家用机器人不再需要昂贵的实验室数据采集，而是从现有视频库里“零成本”提取技能，让机器人学做菜、整理家务的门槛大幅降低。虽然目前还处于研究阶段，但Pegasus给具身智能指明了一条新路：与其让机器人从零摸索，不如把人类已有的经验大规模“迁移”给它们，这或许能让家用机器人更快走进普通人的日常生活。

arXiv:2607.26903v1 Announce Type: new Abstract: The key bottleneck in embodied AI is not model architecture but data. Although billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap between human morphology and robot hardware. We introduce Pegasus, a low-resource framework that bridges this gap by translating human demonstrations into robot-learnable data through structured knowledge transfer. Instead of relying on raw video prompts, Pegasus constructs a graph-based intermediate representation: a Task Graph extracted from human videos is transformed through Affordance and Constraint Graphs into a Robot Planning Graph for robot-conditioned video generation. A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, enabling generalization beyond object identities. A closed-loop physics verifier further filters invalid generations using kinematic feasibility, collision constraints, and joint limits. We evaluate Pegasus across a range of egocentric manipulation benchmarks, including GTEA Gaze+ and EPIC-KITCHENS-100, and diverse robot embodiments, assessing Task Correctness, Executability, State Consistency, and Learnability. Results demonstrate reliable cross-embodiment translation and show that robot data generation can be reframed from a hardware collection problem into a scalable, low-resource knowledge transfer problem.