Purpose To externally validate our surgical skill assessment system, which provides comprehensive real-time feedback based on motion capture (Mocap) metrics of laparoscopic instruments in simulation training environments. Methods Participants conducted periaortic tissue dissection (Dissection Task) and parenchymal closure of the kidney (Suturing Task) using porcine cadaveric organs. Surgical instrument movements were measured by the MoCap system, and all training sessions were video-recorded. Utilizing our previously developed machine-learning (ML) algorithm, participants received quantitative feedback based on their motion features immediately after the training, including three-tier classification results, which meant their skill levels being consistent with those with <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="464_2025_12018_Article_IEq1.gif" Format="GIF" Height="15" Rendition="HTML" Resolution="72" Type="Linedraw" Width="31" /> </InlineMediaObject> <EquationSource Format="TEX">$0 \le$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>0</mn> <mo>≤</mo> </mrow> </math></EquationSource> </InlineEquation> to ≤ 9 surgical caseloads of laparoscopy (novice), 10 <InlineEquation ID="IEq2"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="464_2025_12018_Article_IEq2.gif" Format="GIF" Height="15" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">$\le$</EquationSource> <EquationSource Format="MATHML"><math> <mo>≤</mo> </math></EquationSource> </InlineEquation> to ≤ 49 (intermediate), and <InlineEquation ID="IEq3"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="464_2025_12018_Article_IEq3.gif" Format="GIF" Height="15" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">$\ge$</EquationSource> <EquationSource Format="MATHML"><math> <mo>≥</mo> </math></EquationSource> </InlineEquation> 50 (experienced), and predicted Global Operative Assessment of Laparoscopic Skills (GOALS) scores. Automatically generated feedback comments were also provided through explainable artificial intelligence technology, and participants’ satisfaction with this skill assessment system was also recorded. For the three-tier classifications, accuracy was calculated by comparison between estimated and actual surgical experience levels. The correlation and mean absolute error (MAE) were calculated between the algorithm-predicted and expert-evaluated mean GOALS scores. Results A total of 38 urologists, 4 junior residents, and 10 medical students voluntarily participated in the current training. Several participants repeatedly performed the training, and so 107 datasets were included in the present analysis. Validation analyses demonstrated overall classification accuracies of 67.3 and 56.9% for Dissection Task and Suturing Task, respectively, when excluding the subsequent data of repeaters. In terms of GOALS score prediction, the correlation coefficient and MAE were 0.86 and 2.51 <InlineEquation ID="IEq4"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="464_2025_12018_Article_IEq4.gif" Format="GIF" Height="13" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">$\pm$</EquationSource> <EquationSource Format="MATHML"><math> <mo>±</mo> </math></EquationSource> </InlineEquation> 1.82 for Dissection Task, and 0.91 and 1.56 <InlineEquation ID="IEq5"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="464_2025_12018_Article_IEq4.gif" Format="GIF" Height="13" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">$\pm$</EquationSource> <EquationSource Format="MATHML"><math> <mo>±</mo> </math></EquationSource> </InlineEquation> 1.20 for Suturing Task, respectively, when compared with experts’ evaluations. Overall, 88% of the participants found the feedback easy to understand, and 75% of participants were satisfied with the evaluation. Conclusions Our real-time feedback system was viewed favorably by the participants, and the ML algorithm was a reliable method, especially the predicted GOALS score. This system could facilitate objective performance assessment, enhance simulation-based training efficacy, and potentially accelerate surgical skill acquisition.

External validation of a motion capture-based surgical skill assessment system in laparoscopic simulation training environments

期刊论文
出版模式：

开放获取
发表日期：

2025年7月26日

查看全文

Koki Ebina,
Takashige Abe,
Kiyohiko Hotta,
Yan Lingbo,
Chihiro Kamijo,
Madoka Higuchi,
Masafumi Kon,
Hiroshi Kikuchi,
Haruka Miyata,
Ryuji Matsumoto,
Takahiro Osawa,
Sachiyo Murai,
Taku Senoo,
Shunsuke Komizunai,
Teppei Tsujita,
Kazuya Sase,
Xiaoshuai Chen,
Atsushi Konno,
Nobuo Shinohara

摘要

Purpose

To externally validate our surgical skill assessment system, which provides comprehensive real-time feedback based on motion capture (Mocap) metrics of laparoscopic instruments in simulation training environments.

Methods

Participants conducted periaortic tissue dissection (Dissection Task) and parenchymal closure of the kidney (Suturing Task) using porcine cadaveric organs. Surgical instrument movements were measured by the MoCap system, and all training sessions were video-recorded. Utilizing our previously developed machine-learning (ML) algorithm, participants received quantitative feedback based on their motion features immediately after the training, including three-tier classification results, which meant their skill levels being consistent with those with $0 \le$ $0 \leq$ to ≤ 9 surgical caseloads of laparoscopy (novice), 10 $\le$ $\leq$ to ≤ 49 (intermediate), and $\ge$ $\geq$ 50 (experienced), and predicted Global Operative Assessment of Laparoscopic Skills (GOALS) scores. Automatically generated feedback comments were also provided through explainable artificial intelligence technology, and participants’ satisfaction with this skill assessment system was also recorded. For the three-tier classifications, accuracy was calculated by comparison between estimated and actual surgical experience levels. The correlation and mean absolute error (MAE) were calculated between the algorithm-predicted and expert-evaluated mean GOALS scores.

Results

A total of 38 urologists, 4 junior residents, and 10 medical students voluntarily participated in the current training. Several participants repeatedly performed the training, and so 107 datasets were included in the present analysis. Validation analyses demonstrated overall classification accuracies of 67.3 and 56.9% for Dissection Task and Suturing Task, respectively, when excluding the subsequent data of repeaters. In terms of GOALS score prediction, the correlation coefficient and MAE were 0.86 and 2.51 $\pm$ $\pm$ 1.82 for Dissection Task, and 0.91 and 1.56 $\pm$ $\pm$ 1.20 for Suturing Task, respectively, when compared with experts’ evaluations. Overall, 88% of the participants found the feedback easy to understand, and 75% of participants were satisfied with the evaluation.

Conclusions

Our real-time feedback system was viewed favorably by the participants, and the ML algorithm was a reliable method, especially the predicted GOALS score. This system could facilitate objective performance assessment, enhance simulation-based training efficacy, and potentially accelerate surgical skill acquisition.