A Personalized Captioning Strategy for the Deaf and Hard-of-Hearing Users in an Augmented Reality Environment

Integrating Augmented Reality (AR) in learning environments has revolutionized traditional learning approaches by providing immersive and interactive experiences beyond conventional constraints. While several studies have used AR to enhance learning for individuals with disabilities, there is a significant gap in research focusing on generating captions within AR environments to optimize the experience for users who are deaf or hard of hearing (DHH). This study aims to investigate a personalized captioning strategy tailored to the unique needs of the DHH community. To achieve this, a scoping review of published articles in three databases was conducted to discover how captions are generated, where they are placed, and what attributes of captions are personalizable in AR spaces. Using the PRISMA methodology followed by an interview, the study identified nomenclature discrepancies in AR spaces for hand-held devices, proposing the terms “world-lock-view” and “screen-lock-view”. While analyzing captioning technology, the study identified a lack of customization options in Automatic Speech Recognition implementations. This is particularly significant for DHH users in AR spaces, who require control over font styles, background settings, and caption placement to enhance accessibility for DHH users within AR environments, contributing to a more inclusive and enriched learning experience. The study also discusses the limitations of the captioning strategy and proposes solutions to these limitations.