Towards Fair Detection of AI-Generated Essays in Large-Scale Writing Assessments

The release of ChatGPT in late 2022 triggered revolutionary advances in generative artificial intelligence (AI). With the increasing adoption of these AI tools for generating high-quality texts in response to writing assignments or assessments in educational settings, it is imperative to detect AI-generated essays to maintain academic integrity and test security. Although several automated detectors targeting AI-generated texts have been developed, an effective detector must not only exhibit high detection accuracy but also not display bias across demographic groups. In this study, we compare the effectiveness of three strategies—balancing training data, removing sensitive features, and adjusting thresholds—in mitigating the detector bias concerning native and non-native English speakers in the context of a large-scale writing assessment. Leveraging a dataset comprised of 85,567 essays, our results show that these strategies can reduce the bias in detecting AI-generated essays to varying degrees without significantly compromising detection accuracy.