Towards Reliable E2R Texts: A Proposal for Standardized Evaluation Practices

Easy-to-Read (E2R) is a method of enhancing the accessibility of written text by using clear, direct, and simple language. E2R texts are designed to improve readability and accessibility, especially for individuals with cognitive disabilities. However, there is a significant lack of standardized evaluation methods for these texts. Traditional Automatic Text Simplification (ATS) evaluation methods such as BLEU, SARI or ROUGE present several limitations for E2R evaluation. Readability measures such as Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKGL) do not take into account all document factors. Manual evaluation methods, such as Likert scales, are resource-intensive and lead to subjective assessments. This paper proposes a threefold evaluation method for E2R texts. The first step is an automatic evaluation to measure quantitative aspects related to text complexity. The second step is a checklist-based manual evaluation that takes into account qualitative aspects. The third step is a user evaluation, focusing on the needs of end-users and the understandability of texts. Our methodology ensures thorough assessments of E2R texts, even when user evaluations are not feasible. This approach aims to bring standardization and reliability to the evaluation process of E2R texts.