Purpose Intraoperative speech monitoring during awake glioma surgery is critical to detect stimulation-induced motor speech impairments such as dysarthria. However, automatic detection from short speech segments remains challenging due to limited annotations, strong acoustic variability, and transient impairments in operating-room conditions. This study investigates whether combining complementary encodings of the same log-mel representation can improve robustness under such constraints. Methods We propose a dual-branch EfficientNet-B0 architecture that processes two complementary views of each log-mel spectrogram: (i) a numeric amplitude-preserving representation stored as NPY, and (ii) an RGB spectrogram image stored as PNG using a fixed colormap. The resulting embeddings are fused using four strategies: naïve concatenation, attention-based fusion, and gated fusion (sigmoid and GRU-style). Models are evaluated under two settings: (i) intra-corpus group-wise cross-validation on TORGO (English, controlled) and DATABRASE (French, intraoperative), and (ii) cross-corpus transfer (TORGO<InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\rightarrow $</EquationSource> <EquationSource Format="MATHML"><math> <mo stretchy="false">→</mo> </math></EquationSource> </InlineEquation>DATABRASE) to quantify domain shift. In addition, we evaluate DATABRASE with and without Demucs-based denoising to analyze the impact of operating-room noise. Results In intra-corpus evaluation, adaptive fusion strategies provide consistent benefits on TORGO, with gated fusion achieving the highest AUC (0.852±0.087). On raw DATABRASE, the PNG unimodal baseline yields the best mean performance (UAR 0.830±0.132; AUC 0.889±0.109), whereas denoising altered the model ranking and improved the performance of attention-based fusion (UAR 0.854±0.106). Cross-corpus testing reveals a marked degradation for all methods, confirming a severe domain shift between TORGO and intraoperative speech. Conclusion Dual-encoding fusion can enhance dysarthria detection when training and testing conditions are aligned and noise is controlled, but robust cross-corpus generalization remains unresolved. These results highlight both the potential and current limitations of spectrogram-based deep models for intraoperative speech monitoring, motivating future work on domain-adaptive and self-supervised learning tailored to clinical recordings.

Dual-view fusion of log-mel spectrogram encodings for dysarthria detection: use in the context of awake brain surgery

期刊论文
出版模式：

开放获取
发表日期：

2026年7月10日

查看全文

Nassib Abdallah,
Harrison Misy,
Jean-Marie Marion,
Chinmayi Kanthila,
Celine Panheleux,
Vanessa Saliou,
Romuald Seizeur,
Guillaume Dardenne

摘要

Purpose

Intraoperative speech monitoring during awake glioma surgery is critical to detect stimulation-induced motor speech impairments such as dysarthria. However, automatic detection from short speech segments remains challenging due to limited annotations, strong acoustic variability, and transient impairments in operating-room conditions. This study investigates whether combining complementary encodings of the same log-mel representation can improve robustness under such constraints.

Methods

We propose a dual-branch EfficientNet-B0 architecture that processes two complementary views of each log-mel spectrogram: (i) a numeric amplitude-preserving representation stored as NPY, and (ii) an RGB spectrogram image stored as PNG using a fixed colormap. The resulting embeddings are fused using four strategies: naïve concatenation, attention-based fusion, and gated fusion (sigmoid and GRU-style). Models are evaluated under two settings: (i) intra-corpus group-wise cross-validation on TORGO (English, controlled) and DATABRASE (French, intraoperative), and (ii) cross-corpus transfer (TORGO $\rightarrow $ $\to$ DATABRASE) to quantify domain shift. In addition, we evaluate DATABRASE with and without Demucs-based denoising to analyze the impact of operating-room noise.

Results

In intra-corpus evaluation, adaptive fusion strategies provide consistent benefits on TORGO, with gated fusion achieving the highest AUC (0.852±0.087). On raw DATABRASE, the PNG unimodal baseline yields the best mean performance (UAR 0.830±0.132; AUC 0.889±0.109), whereas denoising altered the model ranking and improved the performance of attention-based fusion (UAR 0.854±0.106). Cross-corpus testing reveals a marked degradation for all methods, confirming a severe domain shift between TORGO and intraoperative speech.

Conclusion

Dual-encoding fusion can enhance dysarthria detection when training and testing conditions are aligned and noise is controlled, but robust cross-corpus generalization remains unresolved. These results highlight both the potential and current limitations of spectrogram-based deep models for intraoperative speech monitoring, motivating future work on domain-adaptive and self-supervised learning tailored to clinical recordings.