Comparative Analysis of Evaluation Measures for Scientific Text Simplification

Access to reliable scientific knowledge is crucial to making informed decisions for both policymakers and citizens. However, scientific documents are inherently difficult due to their complex terminology and vernacular. Automatic text simplification aims to remove some of these barriers. Evaluation frameworks, which include collections and evaluation measures, are designed to assess the generated text simplifications. In this paper, we perform a comparative analysis of current text simplification evaluation measures on both scientific text and a generic corpus based on Wikipedia. Our main finding is that the currently existing measures tend to perform worse on scientific texts and on longer texts consisting of several sentences. More generally, our analysis informs the development of suitable text simplification evaluation measures for scientific texts.