Scholarlink

Purpose <p>Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, safety regulations and operational constraints in operating rooms (ORs) limit agents from freely perceiving and interacting in realistic settings. Digital twins provide high-fidelity, risk-free environments for exploration and training. How we may create photorealistic and dynamic digital representations of ORs that capture relevant spatial, visual, and behavioral complexity remains an open challenge.</p> Methods <p>We introduce <span>TwinOR</span>, a real-to-sim infrastructure for constructing photorealistic and dynamic digital twins of ORs for embodied AI research. The system reconstructs static geometry from pre-scan videos and continuously models human and equipment motion through multi-view perception of OR activities. The static and dynamic components are fused into an immersive 3D environment that supports controllable simulation and facilitates future embodied exploration.</p> Results <p>The proposed framework reconstructs complete OR geometry with centimeter-level accuracy while preserving dynamic interaction across surgical workflows, enabling realistic renderings and a virtual playground for embodied perception benchmarks. In our experiments, <span>TwinOR</span> synthesizes stereo and monocular RGB streams as well as depth observations for geometry understanding and visual localization tasks. Models such as FoundationStereo and ORB-SLAM3 evaluated on <span>TwinOR</span>-synthesized data achieve performance within their reported accuracy ranges on real-world indoor datasets, demonstrating that <span>TwinOR</span> provides sensor-level realism sufficient for emulating real-world perception and localization challenges in dynamic OR scenes.</p> Conclusion <p>By establishing a perception-grounded real-to-sim pipeline, <span>TwinOR</span> enables the automatic construction of dynamic, photorealistic digital twins of ORs. As a safe and scalable environment for experimentation and benchmarking, <span>TwinOR</span> opens new opportunities for translating embodied intelligence from simulation to real-world clinical environments, and sets the stage for future research on interaction, autonomy, and human–robot collaboration in the OR. </p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

TwinOR: photorealistic digital twins of dynamic operating rooms for embodied AI research

  • 期刊论文

  • 发表日期：

    2026年4月16日

查看全文
  • Han Zhang,
  • Yiqing Shen,
  • Roger D. Soberanis-Mukul,
  • Ankita Ghosh,
  • Hao Ding,
  • Lalithkumar Seenivasan,
  • Jose L. Porras,
  • Zhekai Mao,
  • Chenjia Li,
  • Wenjie Xiao,
  • Lonny Yarmus,
  • Angela Christine Argento,
  • Masaru Ishii,
  • Mathias Unberath

摘要

Purpose

Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, safety regulations and operational constraints in operating rooms (ORs) limit agents from freely perceiving and interacting in realistic settings. Digital twins provide high-fidelity, risk-free environments for exploration and training. How we may create photorealistic and dynamic digital representations of ORs that capture relevant spatial, visual, and behavioral complexity remains an open challenge.

Methods

We introduce TwinOR, a real-to-sim infrastructure for constructing photorealistic and dynamic digital twins of ORs for embodied AI research. The system reconstructs static geometry from pre-scan videos and continuously models human and equipment motion through multi-view perception of OR activities. The static and dynamic components are fused into an immersive 3D environment that supports controllable simulation and facilitates future embodied exploration.

Results

The proposed framework reconstructs complete OR geometry with centimeter-level accuracy while preserving dynamic interaction across surgical workflows, enabling realistic renderings and a virtual playground for embodied perception benchmarks. In our experiments, TwinOR synthesizes stereo and monocular RGB streams as well as depth observations for geometry understanding and visual localization tasks. Models such as FoundationStereo and ORB-SLAM3 evaluated on TwinOR-synthesized data achieve performance within their reported accuracy ranges on real-world indoor datasets, demonstrating that TwinOR provides sensor-level realism sufficient for emulating real-world perception and localization challenges in dynamic OR scenes.

Conclusion

By establishing a perception-grounded real-to-sim pipeline, TwinOR enables the automatic construction of dynamic, photorealistic digital twins of ORs. As a safe and scalable environment for experimentation and benchmarking, TwinOR opens new opportunities for translating embodied intelligence from simulation to real-world clinical environments, and sets the stage for future research on interaction, autonomy, and human–robot collaboration in the OR.