Element Extraction from Computer Science Academic Papers for AI Survey Writing

With the exponential growth of research papers, text summarization tools have emerged. However, existing text summarization tools merely extract existing sentences or words based on their frequency and may not be particularly well-suited for papers. To address this gap, this study develops a model based on DistilBERT, primarily focusing on information extraction and dataset labeling and augmentation techniques. The model’s central objective is entity recognition, aiming to identify two specific entities from the full text of research papers. The model takes these critical segments of papers as input and aims to identify the research problems and content contained within them. In response to the limitations of existing datasets, this research augments a dataset with over 4000 full-text arXiv computer algorithm papers through manual annotations. The developed model demonstrates exceptional performance on several evaluation metrics, including accuracy, precision, F1 score, and recall. For comparative experiments, we employed several baseline models based on BERT. These results demonstrate the effectiveness of the proposed model. As part of a comparative experiment, we trained our models using three different dataset training methods. Additionally, to evaluate our dataset’s quality and underline the importance of full-text data, we manually annotated a random selection of 4000 papers from the ARXIV Data dataset, extracting only their titles and abstracts. As a result, Our proposed model outperforms all the baseline models, achieving an accuracy of 0.823 and an F1 Score of 0.798 and models trained on the proposed full-text annotated dataset outperform those trained on other datasets.