Software systems are becoming more and more complex in order to cope with the various complex needs of users. In this context, alerts are often used for analysis to detect faults in complex systems. However, due to the strong correlation between different components of a software system, a system failure typically triggers a large number of alerts. Therefore, reducing the number of alerts becomes a challenging task. This paper leverages pre-trained language models to address this challenge. Semantically related alerts within a short time window are likely to be caused by the same incident. Pre-trained language models can extract deep semantic information from alerts and thus aggregate them belonging to the same incident. This paper conducts a large number of experiments and analyzes on real public alerts datasets, proves the effectiveness of pre-trained language models in the field of alert aggregation, and obtains some interesting findings. Finally, this paper discusses the security of language models and reveals some security issues.