Scholarlink

<p>As the Virtual Reality (VR) industry expands, the need for automated GUI testing is growing rapidly. Large Language Models (LLMs), capable of retaining information long-term and analyzing both visual and textual data, are emerging as a potential key to deciphering the complexities of VR’s evolving user interfaces. In this paper, we conduct a case study to investigate the capability of using LLMs, particularly GPT-4o, for field of view (FOV) analysis in VR exploration testing. Specifically, we validate that LLMs can identify test entities in FOVs and that prompt engineering can effectively enhance the accuracy of test entity identification from <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\varvec{41.67\%}\)</EquationSource> </InlineEquation> to <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\varvec{71.30\%}\)</EquationSource> </InlineEquation>. Our study also shows that LLMs can accurately describe identified entities’ features with at least a <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(\varvec{90\%}\)</EquationSource> </InlineEquation> accuracy rate. We further find out that the core features that effectively represent an entity are color, placement, and shape. Furthermore, the combination of the three features can especially be used to improve the accuracy of determining identical entities in multiple FOVs with the highest F1-score of <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(\varvec{0.70}\)</EquationSource> </InlineEquation>. Additionally, our study demonstrates that LLMs are capable of scene recognition and spatial understanding in VR with precisely designed structured prompts. Finally, we find that LLMs fail to label the identified test entities, and we discuss potential solutions as future research directions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Harnessing large language models for virtual reality exploration testing: a case study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年9月18日

查看全文
  • Zhenyu Qi,
  • Haotang Li,
  • Hao Qin,
  • Kebin Peng,
  • Sen He,
  • Xue Qin

摘要

As the Virtual Reality (VR) industry expands, the need for automated GUI testing is growing rapidly. Large Language Models (LLMs), capable of retaining information long-term and analyzing both visual and textual data, are emerging as a potential key to deciphering the complexities of VR’s evolving user interfaces. In this paper, we conduct a case study to investigate the capability of using LLMs, particularly GPT-4o, for field of view (FOV) analysis in VR exploration testing. Specifically, we validate that LLMs can identify test entities in FOVs and that prompt engineering can effectively enhance the accuracy of test entity identification from \(\varvec{41.67\%}\) to \(\varvec{71.30\%}\) . Our study also shows that LLMs can accurately describe identified entities’ features with at least a \(\varvec{90\%}\) accuracy rate. We further find out that the core features that effectively represent an entity are color, placement, and shape. Furthermore, the combination of the three features can especially be used to improve the accuracy of determining identical entities in multiple FOVs with the highest F1-score of \(\varvec{0.70}\) . Additionally, our study demonstrates that LLMs are capable of scene recognition and spatial understanding in VR with precisely designed structured prompts. Finally, we find that LLMs fail to label the identified test entities, and we discuss potential solutions as future research directions.