MathBERTRes: Mathematical BERT Model for Sequence Classification of Mathematical Research Articles

Analysis of unstructured mathematical text data is a challenging problem. Many pre-trained language models (PLM) based on natural language processing (NLP) are created to comprehend the natural language text. But, the existing PLM ignores the semantic relationship in the mathematical context. In order to extract meaningful insights from the mathematical text documents, a MathBERTRes model (Mathematical Bidirectional Encoder Representations from Transformers model jointly trained with mathematical research articles) has been implemented. The MathBERT model is pretrained with mathematical vocabulary incorporating the mathematical content suitable for all age groups of students from kindergarten to graduate level. The steps involved in the implementation of the proposed system is tokenization, special token inclusion and padding with attention mask, splitting of training and validation data, model training using BERT Sequence Classifier, optimizer creation and learning rate scheduler, model evaluation, and model prediction using test data. The two data sets employed in this paper are verbal arithmetic data set and mathematical research papers. Performance indicators for the system include accuracy and the Matthews correlation coefficient (MCC) score. When verbal arithmetic data is employed, the MathBERT performs exceptionally well with an accuracy of 0.95 and MCC score of 0.898. Similarly, the MathBERTRes also performs well with an accuracy of 0.88 and MCC score of 0.871.