Measuring scientific inquiry ability related to hands-on practice: An automated approach based on multimodal data analysis

Scientific inquiry ability is closely related to the process of hands-on inquiry practice. However, its assessment is often separated from this practice due to the limitation of technical basis and labor cost. The development of multimodal data analysis provides a new opportunity to realize automated assessment based on hands-on practice. Therefore, this study aims to explore whether and how we can use automated multimodal data analysis approaches to measure the scientific inquiry ability of students during the hands-on inquiry practice. In a scientific inquiry activity called "Explore the Moon," designed for 472 fourth graders, we collected textual, tabular, and video data. Aiming to analyze and evaluate the data, we first designed a modal conversion method based on the multimodal pre-trained model LLaVA-7B and a text scoring method integrating keyword matching, one-way nearness, and Jaccard similarity. Then, to bridge the computing ability with the scoring criteria from science teachers, we constructed a structured representation language and verified the human–machine consistency of automated scoring. Finally, we used a multidimensional item response theory (IRT) model to validate the assessment's overall quality and analyze the participants' scientific inquiry ability. The proposed data analysis method has high man–machine consistency, and the results of IRT analysis present reasonable item characteristics. In summary, we constructed a low-cost and scalable multimodal assessment approach based on scientific inquiry activities, providing methodological support for science teachers to carry out formative evaluation of students' scientific inquiry activities in the daily inquiry-based learning environment.