Screening conversations often encode signs of psychological distress not only in lexical content but also in how utterances are delivered. We study whether large language models (LLMs) can better estimate depression risk when prosodic evidence is made explicit at prompt time. Using the DAIC-WOZ depression subset (AVEC-2017), we construct two prompting pipelines: (i) a transcript-only baseline and (ii) a multimodal variant that aligns 100 Hz COVAREP/formant features to each utterance and converts them into concise natural-language descriptors (e.g., low pitch, flat tone, long pauses). We keep the protocol constant across systems (official train/dev splits, PHQ-8 risk threshold at 10, and a JSON output schema) and evaluate on the development split. The multimodal prompting consistently improves Macro-F1 and PR-AUC by approximately 0.03 absolute over a strong transcript-only GPT-4 baseline, with parallel reductions in MAE and RMSE for PHQ-8 totals. Ablations show that verbalized acoustics are more effective than appending raw numeric feature summaries alone, and that preserving interviewer PHQ-8 prompts plus the immediate answers yields small but reliable gains. Our results suggest a practical path to multimodal sensitivity without training an audio encoder: align and verbalize prosody so that LLMs can reason over both what was said and how it was said.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Stress Detection with LLM Using DAIC-WOZ

  • Thanh Hai Hoang,
  • Tran Khanh Dang,
  • Duy Nguyen,
  • Nguyen Thi Huyen Trang

摘要

Screening conversations often encode signs of psychological distress not only in lexical content but also in how utterances are delivered. We study whether large language models (LLMs) can better estimate depression risk when prosodic evidence is made explicit at prompt time. Using the DAIC-WOZ depression subset (AVEC-2017), we construct two prompting pipelines: (i) a transcript-only baseline and (ii) a multimodal variant that aligns 100 Hz COVAREP/formant features to each utterance and converts them into concise natural-language descriptors (e.g., low pitch, flat tone, long pauses). We keep the protocol constant across systems (official train/dev splits, PHQ-8 risk threshold at 10, and a JSON output schema) and evaluate on the development split. The multimodal prompting consistently improves Macro-F1 and PR-AUC by approximately 0.03 absolute over a strong transcript-only GPT-4 baseline, with parallel reductions in MAE and RMSE for PHQ-8 totals. Ablations show that verbalized acoustics are more effective than appending raw numeric feature summaries alone, and that preserving interviewer PHQ-8 prompts plus the immediate answers yields small but reliable gains. Our results suggest a practical path to multimodal sensitivity without training an audio encoder: align and verbalize prosody so that LLMs can reason over both what was said and how it was said.