From Sensory Perception to Realtime NonVerbal Communication

Researchers have found challenges regarding social perception in people with autism. Specific areas of inquiry are joint attention, that involves following an interaction by tracking eye movement (Noel et al., IEEE Transactions on Cognitive and Developmental Systems 10:973–982, 2018), understanding paralinguistics cues such as the voice, dynamic facial expressions (Mundy et al., Journal of Autism and Developmental Disorders 17:349–364, 1987), and integrating audio with visual modalities in speech (Belin et al., Trends in Cognitive Sciences 8:129–135, 2004). Many assistive technologies that aim to support neurodivergence focus on behavior change, remediation, or skill acquisition to modify non-normative behavior. This is especially true in that realm of social skills training and includes my early work (L. E. Boyd, L. E., Rangel, A., Tomimbang, H., Conejo-Toledo, A., Patel, K., Tentori, M., & Hayes, G. R. (2016). SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 4872–4883. https://doi.org/10.1145/2858036.2858215 ; L. E. Boyd, Jiang, et al., 2017; L. E. Boyd, Ringland, et al., 2017; L. E. Boyd, L. E., Gupta, S., Vikmani, S. B., Gutierrez, C. M., Yang, J., Linstead, E., & Hayes, G. R. (2018a). vrSocial: Toward Immersive Therapeutic VR Systems for Children with Autism. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–12. https://doi.org/10.1145/3173574.3173778 ) as I transition from a clinician to an assistive technologist (Du, Y., Boyd, L., & Ibrahim, S. (2018). From Behavioral and Communication Intervention to Interaction Design: User Perspectives from Clinicians. 6.). It has been reported that most technologies to date have focused on ways to coach or correct undesired behavior (Spiel, K., Frauenberger, C., Keyes, O., & Fitzpatrick, G. (2019). Agency of Autistic Children in Technology Research: A Critical Literature Review. ACM Transactions on Computer–Human Interaction, 26(6), 38:1–38:40. https://doi.org/10.1145/3344919 ). For example, (Noel et al., IEEE Transactions on Cognitive and Developmental Systems 10:973–982, 2018) demonstrated a delay in processing real time social cues and responding with nonverbal cues. This chapter discusses the importance of supporting the dynamic aspects of nonverbal communication in technology design as well as delineates design principles to support more comfortable and accessible social communication.