A study on user Danmaku sending and emotional expression in online travel videos based on multimodal fusion

Based on stimulus–response (S–R) theory, this study aims to investigate the impact of the interaction of textual, acoustic, and visual features of online travel videos on viewers’ engagement in sending Danmaku comments and expressing emotions. We collected 130 short videos from travel bloggers on the Bilibili and segmented them into 1024 video clips. Our study employed the Low-rank multimodal fusion (LMF) fusion strategy model to conduct the ablation study and the series of experimental results from the fusion approach indicate that both Danmaku sending and emotional expression are influenced by textual, visual, and auditory information present in micro-videos, but their influence mechanisms exhibit certain distinctions. Specifically, Under the influence of individual modality, visual features have the most prominent impact on Danmaku sending and emotional expression. For the pairwise combination, text + visual and acoustic + visual have the best performance on Danmaku sending, while acoustic + visual has better prediction performance on emotional expression. The combination of three modalities can only maximize the viewers’ emotional expression, not the Danmaku sending behavior. This study thus contributes to engagement behavior in online video platform, considering viewers’ emotional expression behavior, and extends the research on online travel video by showing how multimodal situation influences viewers’ engagement. Suggesting are provided to the video creators as well as platform for online tourism development.