Disentangling the Relationship Between Ability and Test-Taking Effort: To What Extent the Ability Levels Can Be Predicted from Response Behavior?

Exploring the relationship between student ability and test-taking effort is an important area of study, offering insights into their approach to educational assessments. Previous research shows this relationship, yet there is a scarcity of research comparing the test-taking effort of students. In addition, researchers have frequently employed rule-based response time approaches to investigate the test-taking effort after the assessment is completed. This study aimed to address two knowledge gaps in the literature by (1) examining whether students’ abilities differ based on their engagement status in test-taking and (2) developing a predictive model of ability levels through their test-taking effort. Data analysis was completed in three steps using financial literacy assessment in the Program for International Student Assessment (PISA) 2018: (1) conduct response-level filtering and estimate ability scores; (2) use a student-level filtering method to determine the engagement status and run a t-test to determine whether there is a significant difference; and (3) train classifiers for predicting ability levels based on test-taking effort (i.e., response time). t-test results showed a significant difference in ability levels between engaged and disengaged students. This indicated that students who were engaged during the assessment had higher ability scores than those who were not. According to the results from the predictive models, random forest outperformed other classification algorithms. The results suggest that machine learning can assist test administrators or instructors in identifying trends and patterns in the test-taking effort.