TVD-BERT: A Domain-Adaptation Pre-trained Model for Textural Vulnerability Descriptions

Textual Vulnerability Descriptions (TVD) serve as concise natural language summaries within databases like the National Vulnerability Database(NVD), elucidating key facets of software vulnerabilities including impacted products, causation, and impacts. Collecting and analyzing TVD holds significant importance in the field of cybersecurity. In order to better analyze TVD, identify potential trends in software vulnerabilities, and reduce the occurrence of software vulnerabilities, we present a domain-adaptive pre-trained model for textual vulnerability descriptions. First, we collected the most common words in the textual vulnerability descriptions through Term Frequency (TF) method and expanded the BERT vocabulary. Then we adapted the model to textual vulnerability descriptions language (TVDL) and named the trained model TVD-BERT. To further refine the model's adaptability and representational capabilities, we introduce an innovative approach named Double Lexicon Masking, underpinning our Masked Language Modeling (MLM). Our experimental results on tasks related to TVD demonstrate that TVD-BERT can achieve state-of-the-art performance, underscoring its superiority over other methods.