BERTweet.BR: a pre-trained language model for tweets in Portuguese

Recent advancements in neural language models have been primarily centered around English, with limited focus on the more than seven thousand other languages. This includes Portuguese, which, despite being the sixth most spoken language globally, has markedly fewer neural-based linguistic resources available when compared to English. Notably, Portuguese speakers compose one of the most active groups of Twitter users; however, no pre-trained language model in Portuguese tweets is extensively studied in the literature. Besides the language, tweets-based pre-trained models must account for the cultural code, informal linguistic style, code-switching, and the limited number of characters. This manuscript tackles this gap by introducing BERTweet.BR, the first public large-scale pre-trained model specific to the Brazilian Portuguese tweets domain. BERTweet.BR has the same architecture of BERTweet_base, a BERT-based English-tweets model, and was trained from scratch following the RoBERTa pre-training procedure on a 100-M Portuguese tweets corpus. On the sentiment analysis task, experiments show that BERTweet.BR outperforms three multilingual Transformers and BERTimbau, a monolingual general-domain Brazilian Portuguese language model. We release our model in the transformers library aiming at promoting future research in analytical tasks for Portuguese tweets. The BERTweet.BR code, experimental results, and related documentation are publicly available on Github.