MedSum: Medical Video Summarization Using Reinforcement Learning

Medical video summarization is an essential task in healthcare, enabling efficient analysis of lengthy surgical procedures by extracting the most critical and informative content. Traditional methods often struggle to capture the unique temporal and contextual complexities of surgical videos, particularly at the gesture and phase levels. To address these challenges, this paper introduces MedSum, a transformer-based video summarization framework specifically designed for medical applications. MedSum integrates Kernel Temporal Segmentation (KTS) to identify coherent segments and employs transformer-based attention to prioritize significant events, ensuring both informativeness and temporal coherence. Reinforcement learning (RL) drives the frame selection process, optimizing a multi-objective reward function that balances informativeness, coverage, and redundancy. The framework leverages EfficientNet for spatial feature extraction and Farneback optical flow for motion analysis, ensuring robust representation of surgical actions. MedSum has been evaluated on benchmark surgical datasets, JIGSAWS and CATARACTS, demonstrating substantial improvements in F1-Score, Coverage Ratio (CR), and Temporal Smoothness (TS) compared to state-of-the-art methods. It achieves an average F1-Score improvement of over \(18.5\%\) , effectively summarizing both fine-grained gestures and broader procedural phases. By addressing the complexity of surgical workflows, MedSum sets a new benchmark for medical video summarization, while its adaptable design holds promise for broader applications across healthcare and other video-intensive domains.