FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

AI 中文解读

核心亮点是，科学家专门为拥挤混乱的全球南方城市打造了一个AI“世界模拟器”，让AI能更准确地预判街头下一秒会发生什么。通俗地说，过去AI预测场景主要靠的是欧美那种车道分明、行人稀疏的环境。但现实中的很多城市，街巷拥挤、人车混行、遮挡严重，AI常常“看不过来”。这项研究首次收录了22个城市、1000小时的街景视频，并设计了一个新模型，把“谁在哪、在干嘛、怎么互动”分开处理，即使只看到一小截画面，也能更聪明地预测后续动态。实验证明，它的预测精度和抗干扰能力都明显提升。普通人的生活会受到什么影响呢？最直接的就是自动驾驶和安防监控。以后无人车在混乱路口更知道提前刹车，监控系统也能在人群密集时更快发现异常。更长远看，从外卖配送机器人到城市管理规划，这套技术都能让机器更懂真实世界的复杂性，而不是只适应“样板间”式的干净街道。目前数据和模型都已公开，意味着更多开发者和研究者可以在此基础上继续改进，离落地又近了一步。

World models have attracted significant attention for their ability to capture and predict the structure and dynamics of the physical world. In this emerging landscape, Joint Embedding Predictive Architectures (JEPA) offer a particularly compelling direction. We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments, which we call DENSEWORLD. Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft spatial boundaries, extreme agent heterogeneity, persistent occlusion, and rapid social negotiation under mixed traffic. We introduce the first large-scale dataset for this regime: 1,000 hours of drive-through, walk-through, and aerial video across 22 cities. Existing JEPA formulations struggle to preserve dense interaction dynamics under heterogeneity and partial observability. We introduce FactorJEPA, which makes world structure a first-class predictive primitive. Rather than encoding the future in a monolithic latent, it composes layout, entities, and interactions, using a visibility gate and separated subspaces to preserve partially observed agents and discourage cross-factor shortcuts. FactorJEPA improves (i) future-latent accuracy (Future-frame L1), (ii) intervention-sensitive prediction (Causal L1), and (iii) robustness to reduced visual evidence (Mask-ratio slope), while exposing (iv) a reproducible motion-information trade-off (Motion cosine). Method rankings replicate across 2B and 1B V-JEPA 2.1 backbones, with rho = 0.895 to 0.978. We publicly release the DENSEWORLD-115k dataset (https://huggingface.co/datasets/anonymousML123/denseworld-115k) and the surgery-trained FactorJEPA checkpoints (https://huggingface.co/datasets/anonymousML123/factorjepa-outputs/tree/main/outputs/full/vjepa_2_1_vitg_1B/train/m09c_surgery_3stage_DI_diheavy_encoder).