PPTopicPLM: plug-and-play topic-enhanced pre-trained language model for short-text rumor detection

Recently, lots of pre-trained language models (PLMs) have been investigated for rumor detection and obtained superior results. However, existing PLM-based approaches are challenged and limited when addressing short texts which are often sparse, noisy, and semantically ambiguous. To remedy this, we propose a Plug-and-Play Topic-enhanced Pre-trained Language Model for short-text rumor detection, dubbed PPTopicPLM, aiming to combine topic models with PLMs without modifying the core network structure of PLMs. First, the pre-trained RoBERTa is used to represent the short texts. Then, we utilize the semantics-assisted non-negative factorization (SeaNMF) model to explore topics in short texts since SeaNMF has proven to be effective in incorporating the word-context semantic correlations. Instead of simple but computationally expensive methods like word frequency statistics, we build the semantic correlation matrix based on the semantic similarity between the dynamic word embeddings yielded by pre-trained PLMs to mitigate the semantic information loss especially encountered by short texts. Afterward, we employ the multimodal factorized bilinear pooling technique to thoroughly aggregate topics and text semantics. Finally, substantial experiments conducted on four real-world datasets show that the proposed method achieves significant performance improvements due to its successfully joining topic models and PLMs for short-text rumor detection. We further empirically demonstrate the wide application prospect of our approach on the semantic similarity detection task. The codes are open sourced at https://github.com/zjfgh2015/PPTopicPLM.