Towards Improving the Correct Lyric Detection by Deaf and Hard of Hearing People

Access to music for people who are Deaf/Hard of Hearing (D/HoH) includes the instrumental portion and the lyrics. While closed captioning can provide the lyrics in text format, it does not necessarily provide accurate timing of the lyrics with the instrumental portion. This study aims to clarify how vibrotactile stimuli affect understanding the onset timing of song lyrics for D/HoH people. To achieve this goal, we developed a system called VIBES: VIBrotactile Engagement for Songs that simultaneously provides music and vibration playback as an iPhone app. Unlike other vibrotactile systems for music, which focus primarily on the percussion/beat or frequencies of the instrumental portions, VIBES presents vibrations for the timing of vocal utterances, syllable by syllable. We conducted a study with 10 participants to determine the system’s effectiveness by comparing the understanding of lyric timing with vibration and sound only. Statistically, one of the four songs in the experiment showed significant differences, where understanding lyric timing with vibration is better than in sound-only conditions. Two songs revealed the opposite results, and the other showed no differences between the two conditions. Although these results were unexpected, we obtained several findings to make the next version of VIBES, such as the frequency of vibrations.