Evaluating the reliability of remote video based assessment metrics for the advanced training in laparoscopic suturing (ATLAS) curriculum–a pilot study

The Advanced Training in Laparoscopic Suturing (ATLAS) curriculum was created to address gaps in advanced laparoscopic suturing training and has been previously studied using in-person assessment (IPA). This study aimed to evaluate the reliability of a remote, asynchronous video-based assessment (VBA) protocol by comparing to IPA to build validity data for the ATLAS curriculum.

Three surgeons (two trainees, one expert) at a single institution performed five repetitions of each ATLAS task. Each performance underwent IPA by a single rater and remote VBA by two independent raters. Videos were de-identified and randomized prior to scoring. Task scores were calculated using time and error measurements per the ATLAS scoring rubric. The VBA scores were averaged and compared to IPA scores by calculating intraclass correlation coefficients (ICC).

A total of 90 task performances were reviewed. A higher number of errors were detected in VBA than in IPA (393.5 vs. 323) with higher detection frequency in nine of fifteen errors. Average VBA scores were lower than average IPA scores across all tasks, but with moderate-to-excellent reliability for each task; the ICC of all tasks combined was 0.93 (P < 0.001, 95% CI [0.89, 0.96]).

All ATLAS tasks were found to have moderate to excellent reliability between IPA and VBA for task scores but with an increased frequency of error detection in VBA. VBA of ATLAS is reliable and may improve adoption of this curriculum by minimally invasive and general surgery trainees. Future studies are needed to further evaluate the use of VBA in ATLAS and other skills curricula.