The Role of First Language in Automated Essay Grading for Second Language Writing

The advancement of AI has paved the way for human-like performance in numerous tasks. One field that has seen significant benefits from its application is the automated essay grading (AEG). Recent AEG systems have demonstrated impressive accuracy in evaluating essay quality. However, the majority of AEG systems have been developed with a primary focus on essays produced in a first language (L1) context although there are countless second language learners around the world who need their writing skills assessed for education or evaluation reasons. More importantly, most AEG systems for L2 essays do not consider the impact of L1 which is crucial in shaping linguistic features within them, possibly limiting their applicability to some essays written by learners whose L1 is not given adequate consideration. This, in turn, may bring up concerns about the equal representation and inclusion of essays from learners with various L1 backgrounds. To investigate this possibility, we developed 11 AEG systems using the XGBoost algorithm, each based on the essay data of a specific L1 group (Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, Turkish), and compared these systems’ performance. Furthermore, we ascertained the relative importance of linguistic features in each system. Our study showed that the AEG systems provided with essay data from different L1 groups varied in accuracy, with slightly different linguistic features being the most crucial in the system. This suggests the necessity to include more diverse and inclusive data from learners with different L1 backgrounds in the development of AEG systems.