Multi-token Fusion Framework for Multimodal Sentiment Analysis

In this paper, we design a multi-token fusion (MTF) framework to process inter-modality and intra-modality information in parallel for multimodal sentiment analysis. Specifically, a tri-token transformer (TT) module is proposed to extract three tokens from each modality where one of them retains the unimodal feature and the other two tokens learn multi-modal features from the other two modalities respectively. Furthermore, a module based on the hierarchical element-wise self-attention (HESA) is used to process the three tokens of each modality extracted by TT. As a result, the important elements of tokens will be given more attention. Finally, we conduct extensive experiments on two public datasets, which prove the effectiveness and scalability of our network.