We present a standardized protocol and dataset design for multimodal acquisition in older adults with suspected cognitive impairment, aimed at supporting early identification research and data-intensive machine learning. The protocol combines four time-bounded discourse elicitation tasks, personal narrative, picture description, story narration, and procedural discourse, to capture complementary linguistic and cognitive demands. Speech is recorded with an external microphone and transcribed to provide aligned audio–text modalities, while an RGB camera facing the participant records facial behavior to enable analysis of nonverbal cues. In addition, we integrate an immersive VR/MR serious-game module inspired by our HoloDemTect framework, including daily-life activities such as a shopping-list task that requires selecting target items and placing them into a box. The VR/MR environment is fully instrumented to retain fine-grained telemetry (e.g., completion times, step-level latencies, interaction events, and error patterns), and gaze-related measures are recorded when supported by the hardware. To provide clinically meaningful reference outcomes for operational stratification and downstream classification, each session includes a brief self-assessment of cognition and instrumental functioning using Test Your Memory and the Lawton & Brody IADL scale. Overall, the protocol yields a structurally complete and diverse dataset that enables joint analyses of speech, language, facial behavior, and immersive task performance, addressing common unimodality constraints in existing resources and facilitating multimodal deep learning baselines and future benchmarking.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Standardized Multimodal Dataset Recording Protocol for Early Cognitive Impairment Screening

  • David Ortiz-Perez,
  • David Mulero-Pérez,
  • David Alarcon-Garrido,
  • Laura Saval-Cillero,
  • Jose Garcia-Rodriguez,
  • M. Flores Vizcaya-Moreno

摘要

We present a standardized protocol and dataset design for multimodal acquisition in older adults with suspected cognitive impairment, aimed at supporting early identification research and data-intensive machine learning. The protocol combines four time-bounded discourse elicitation tasks, personal narrative, picture description, story narration, and procedural discourse, to capture complementary linguistic and cognitive demands. Speech is recorded with an external microphone and transcribed to provide aligned audio–text modalities, while an RGB camera facing the participant records facial behavior to enable analysis of nonverbal cues. In addition, we integrate an immersive VR/MR serious-game module inspired by our HoloDemTect framework, including daily-life activities such as a shopping-list task that requires selecting target items and placing them into a box. The VR/MR environment is fully instrumented to retain fine-grained telemetry (e.g., completion times, step-level latencies, interaction events, and error patterns), and gaze-related measures are recorded when supported by the hardware. To provide clinically meaningful reference outcomes for operational stratification and downstream classification, each session includes a brief self-assessment of cognition and instrumental functioning using Test Your Memory and the Lawton & Brody IADL scale. Overall, the protocol yields a structurally complete and diverse dataset that enables joint analyses of speech, language, facial behavior, and immersive task performance, addressing common unimodality constraints in existing resources and facilitating multimodal deep learning baselines and future benchmarking.