Scholarlink

<p>Visual-language foundation models (VLFMs) are rapidly emerging as key enablers of interaction in virtual and augmented reality (VR/AR). By integrating advanced visual perception with language-based reasoning, they overcome the limitations of rule-driven systems and enable more natural, adaptive, and context-aware engagement. This paper provides a systematic survey of the evolving roles of VLFMs in immersive environments. We introduce the Interaction-Role Taxonomy, a novel framework that categorizes VLFMs into three complementary functions: the Semantic Interpreter, which perceives and interprets environments; the Embodied Agent, which facilitates collaborative interaction; and the Generative Engine, which creates dynamic content. Building on this framework, we review representative applications, analyze implementation pipelines and evaluation protocols, and critically discuss technical and ethical challenges. Finally, we identify open research directions to enhance the capability, efficiency, and trustworthiness of VLFMs, paving the way for next-generation human–computer interaction in immersive VR/AR ecosystems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A survey of visual-language foundation models for enhancing virtual and augmented reality interactivity

  • 期刊论文

  • 发表日期：

    2026年1月13日

查看全文
  • Jingxuan Wang,
  • Weiliang Meng,
  • Yixiao Jin,
  • Jinyuan Wang,
  • Yiming Qin,
  • Meng Yang

摘要

Visual-language foundation models (VLFMs) are rapidly emerging as key enablers of interaction in virtual and augmented reality (VR/AR). By integrating advanced visual perception with language-based reasoning, they overcome the limitations of rule-driven systems and enable more natural, adaptive, and context-aware engagement. This paper provides a systematic survey of the evolving roles of VLFMs in immersive environments. We introduce the Interaction-Role Taxonomy, a novel framework that categorizes VLFMs into three complementary functions: the Semantic Interpreter, which perceives and interprets environments; the Embodied Agent, which facilitates collaborative interaction; and the Generative Engine, which creates dynamic content. Building on this framework, we review representative applications, analyze implementation pipelines and evaluation protocols, and critically discuss technical and ethical challenges. Finally, we identify open research directions to enhance the capability, efficiency, and trustworthiness of VLFMs, paving the way for next-generation human–computer interaction in immersive VR/AR ecosystems.