Supporting the Communication of People with Aphasia While Lying in Bed

Communication is an essential part of life. Therefore, when affected due to an acquired language impairment, such as Aphasia, it has a negative impact on the person’s quality of life. Although Augmentative and Alternative Communication (AAC) solutions are available, they are generally not convenient for all contexts, such as when the person is lying in bed, where existing solutions can be difficult to reach/use. In this context, we propose a system supporting communication for people with Aphasia (PWAs) by enabling communication between a PWA lying in bed, alone, and a remote person (e.g., caregiver). The communication is mediated by an assistant asking the PWA simple Yes/No questions, presented in a multimodal way, which can be answered through gestures. To design, validate, and refine our first system proposal, we engaged in discussions with Speech and Language Therapists (SLTs) who have experience with PWAs. Their feedback emphasized the proposed system’s adequateness and its strong potential to increase PWAs’ independence. The outcomes of the discussion with SLTs also led to the selection of two gestures for input by the PWAs: knocking on the mattress for “yes” and the movement as if cleaning the mattress for “no”. To establish the technical viability of using these gestures, we then evaluated a gesture recognition approach based on smartwatch sensor data collected from eight subjects. For a user-independent solution, a relatively good performance was achieved using the random forest algorithm: overall accuracy and F1 score of 86%, false positive rate \(\approx \) 0%, and arm gesture accuracy \(\approx \) 100%.