A Speech-Driven Embodied Listening System with Mirroring of Pupil Response

It is known that human pupils are related to own interest and motivation, and the pupil dilation increases attractiveness. Focusing on the effect of pupil response, we analyzed the relationship between a speaker’s speech and his/her own pupil response, and confirmed that pupils dilate in synchronization with speech. Furthermore, we developed a pupil response system that expresses various pupil responses and demonstrated its effectiveness. In particular, the pupil dilation can convey strong interest and high motivation to the speaker. Therefore, it is expected to develop an advanced system that actively engages the speaker by expressing pupil response as a listener. In this study, we developed a speech-driven listening system that mirrors the pupil response in synchronization with the speaker’s speech. This system can express positive listening attitudes such as pupil responses and nodding responses. The effectiveness of this system was demonstrated by sensory evaluations.