Empirical Analysis of Individual Differences Based on Sentiment Estimation Performance Toward Speaker Adaptation for Social Signal Processing

Understanding user’s internal state is indispensable for human-robot interaction in social signal processing. To mitigate the bias of sentiments observed by third-party annotators, the importance of self-reported by users themselves was pointed out recently. However, the self-reported internal state is not displayed as similar multimodal behaviors among different individuals, this leads to performance gap between self-reported and third-party sentiment estimations. Speaker adaptation for social signal processing (SASSP) is necessary to learn individual social signal characteristics to mitigate the individual differences. Towards effective adaptation for speakers with different characteristics, clarifying influence of individual differences in internal state estimation is necessary but has not been clarified. To address this problem, this study conducted empirical analysis by training and testing models on multimodal data of a group of speakers. Then, we analyze the relationships between the best model’s performance and speaker’s characteristics including age, gender, personalities, and speaker’s expectation before human-robot interaction experiment. The results showed that these aspects all have influence on estimation performance in SASSP due to expression differences. This study provides suggestions and directions on setting SASSP policies for self-reported internal state estimation.