Multilingual Transformer and BERTopic for Short Text Topic Modeling: The Case of Serbian

This paper presents the results of the first application of BERTopic, a state-of-the-art topic modeling technique, to short text written in a morphologically rich language. We applied BERTopic with three multilingual embedding models on two levels of text preprocessing (partial and full) to evaluate its performance on partially preprocessed short text in Serbian. We also compared it to LDA and NMF on fully preprocessed text. The experiments were conducted on a dataset of tweets expressing hesitancy toward COVID-19 vaccination. Our results show that with adequate parameter setting, BERTopic can yield informative topics even when applied to partially preprocessed short text. When the same parameters are applied in both preprocessing scenarios, the performance drop on partially preprocessed text is minimal. Compared to LDA and NMF, judging by the keywords, BERTopic offers more informative topics and gives novel insights when the number of topics is not limited. The findings of this paper can be significant for researchers working with other morphologically rich low-resource languages and short text.