Purpose <p>Intraoperative feedback is essential to surgical training, yet it is transient, inconsistently documented, and difficult to analyze at scale. Traditional qualitative methods for studying operative dialogue are labor-intensive, limiting their use for educational research and surgical trainee feedback. This study describes the development and evaluation of an AI-assisted qualitative coding pipeline for analyzing intraoperative attending-trainee communication. We frame this work as a proof-of-concept feasibility study.</p> Methods <p>Audio-recorded dialogue from 25 general and colorectal surgical operations was transcribed and formatted for line-by-line coding using a 38-category thematic codebook. A custom GPT built on GPT-4o with retrieval-augmented generation was developed to assign codes to each utterance within a human-in-the-loop workflow. AI reliability was assessed through run-to-run agreement on a purposive sample of five transcripts. Coding efficiency and error rates were tracked across AI-assisted and manual workflows.</p> Results <p>The AI-assisted workflow reduced coding time to less than 20% of manual effort, with an overall error rate of 2.0%. Run-to-run AI agreement across 2,245 lines demonstrated 79% agreement with substantial concordance (κ = 0.75). Agreement was highest for explicit utterance types and lowest for codes requiring contextual interpretation.</p> Conclusions <p>In this proof-of-concept evaluation, an AI-supported, human-verified coding workflow achieved the efficiency and reliability needed to make scalable analysis of intraoperative teaching dialogue feasible. This approach has the potential to convert routine operative communication into structured, actionable feedback for trainees and faculty.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

From OR dialogue to actionable feedback: a scalable method for analyzing intraoperative teaching

  • Blake T. Beneville,
  • Lauren Lewis,
  • Michael J. Ferzoco,
  • Katharine E. Caldwell,
  • Jenna Bennett,
  • Mohamed A. Jama,
  • Cory Fox,
  • Jonathan Tong,
  • Michael M. Awad

摘要

Purpose

Intraoperative feedback is essential to surgical training, yet it is transient, inconsistently documented, and difficult to analyze at scale. Traditional qualitative methods for studying operative dialogue are labor-intensive, limiting their use for educational research and surgical trainee feedback. This study describes the development and evaluation of an AI-assisted qualitative coding pipeline for analyzing intraoperative attending-trainee communication. We frame this work as a proof-of-concept feasibility study.

Methods

Audio-recorded dialogue from 25 general and colorectal surgical operations was transcribed and formatted for line-by-line coding using a 38-category thematic codebook. A custom GPT built on GPT-4o with retrieval-augmented generation was developed to assign codes to each utterance within a human-in-the-loop workflow. AI reliability was assessed through run-to-run agreement on a purposive sample of five transcripts. Coding efficiency and error rates were tracked across AI-assisted and manual workflows.

Results

The AI-assisted workflow reduced coding time to less than 20% of manual effort, with an overall error rate of 2.0%. Run-to-run AI agreement across 2,245 lines demonstrated 79% agreement with substantial concordance (κ = 0.75). Agreement was highest for explicit utterance types and lowest for codes requiring contextual interpretation.

Conclusions

In this proof-of-concept evaluation, an AI-supported, human-verified coding workflow achieved the efficiency and reliability needed to make scalable analysis of intraoperative teaching dialogue feasible. This approach has the potential to convert routine operative communication into structured, actionable feedback for trainees and faculty.