Scholarlink

The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics understanding. CWM uses a temporally-factored masking policy for masked prediction of video data without annotations. This policy enables highly effective “counterfactual prompting” of the predictor, allowing a spectrum of visual structures to be extracted from a single pre-trained predictor without finetuning on annotated datasets. We demonstrate that these structures are useful for physical dynamics understanding, allowing CWM to achieve the state-of-the-art performance on the Physion benchmark. Code is available at https://neuroailab.github.io/cwm-physics/ .

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Understanding Physical Dynamics with Counterfactual World Modeling

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Rahul Venkatesh,
  • Honglin Chen,
  • Kevin Feigelis,
  • Daniel M. Bear,
  • Khaled Jedoui,
  • Klemen Kotar,
  • Felix Binder,
  • Wanhee Lee,
  • Sherry Liu,
  • Kevin A. Smith,
  • Judith E. Fan,
  • Daniel L. K. Yamins

摘要

The ability to understand physical dynamics is critical for agents to act in the world. Here, we use Counterfactual World Modeling (CWM) to extract vision structures for dynamics understanding. CWM uses a temporally-factored masking policy for masked prediction of video data without annotations. This policy enables highly effective “counterfactual prompting” of the predictor, allowing a spectrum of visual structures to be extracted from a single pre-trained predictor without finetuning on annotated datasets. We demonstrate that these structures are useful for physical dynamics understanding, allowing CWM to achieve the state-of-the-art performance on the Physion benchmark. Code is available at https://neuroailab.github.io/cwm-physics/ .