Purpose Computer-integrated surgical navigation systems are often run in the OR with the assistance of a technician for controlling the user interface and advising on technical details of the system. In lower-resource healthcare settings, limited access to additional OR staff and technical training for operating navigation systems can represent a barrier to sustainably deploying a low-cost surgical navigation solution. Recent advancements in locally deployable large language models (LLMs) have improved their ability to answer technical questions based on source materials and safely perform limited tasks on behalf of a user. Methods The objective of this paper is to explore the feasibility of using a network of local LLM-based agents to act as a natural language interface for a low-cost surgical navigation system, facilitating hands-free manipulation of the user interface and providing documentation-grounded technical guidance. We propose the navigation offline virtual agent (NOVA), an end-to-end architecture that integrates distinct local LLM-based agents for knowledge tasks, action tasks, and delegation between the agents. Two semisynthetic benchmark datasets were generated for ablation studies of individual agents, and a prototype was built which integrates these agents into NousNav, an open-source neuronavigation system. Results The agents based on local LLMs were found to perform comparably to closed-source, commercially hosted LLMs. In a user study with nine participants, NOVA facilitated hands-free patient registration with a mean end-to-end latency of <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$10.4\pm 2.4$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>10.4</mn> <mo>±</mo> <mn>2.4</mn> </mrow> </math></EquationSource> </InlineEquation> s and an 81% command success rate. Conclusions This work demonstrates that local LLM-based agents can be deployed to support users of low-cost navigation systems by providing a context-aware natural language interface, representing an important step toward reducing technician dependence in low-cost surgical navigation.

Local LLMs as cooperative agents for low-cost surgical navigation support

期刊论文
发表日期：

2026年8月2日

查看全文

Colton Barr,
Colin Galvin,
Amirali Azimi,
Sarah Frisken,
Steve Pieper,
Gabor Fichtinger,
Alexandra Golby,
Parvin Mousavi

摘要

Purpose

Computer-integrated surgical navigation systems are often run in the OR with the assistance of a technician for controlling the user interface and advising on technical details of the system. In lower-resource healthcare settings, limited access to additional OR staff and technical training for operating navigation systems can represent a barrier to sustainably deploying a low-cost surgical navigation solution. Recent advancements in locally deployable large language models (LLMs) have improved their ability to answer technical questions based on source materials and safely perform limited tasks on behalf of a user.

Methods

The objective of this paper is to explore the feasibility of using a network of local LLM-based agents to act as a natural language interface for a low-cost surgical navigation system, facilitating hands-free manipulation of the user interface and providing documentation-grounded technical guidance. We propose the navigation offline virtual agent (NOVA), an end-to-end architecture that integrates distinct local LLM-based agents for knowledge tasks, action tasks, and delegation between the agents. Two semisynthetic benchmark datasets were generated for ablation studies of individual agents, and a prototype was built which integrates these agents into NousNav, an open-source neuronavigation system.

Results

The agents based on local LLMs were found to perform comparably to closed-source, commercially hosted LLMs. In a user study with nine participants, NOVA facilitated hands-free patient registration with a mean end-to-end latency of $10.4\pm 2.4$ $10.4 \pm 2.4$ s and an 81% command success rate.

Conclusions

This work demonstrates that local LLM-based agents can be deployed to support users of low-cost navigation systems by providing a context-aware natural language interface, representing an important step toward reducing technician dependence in low-cost surgical navigation.