Traditionally natural language processing (NLP) approaches are used for automated marking of short answers in intelligent tutoring systems. If a classifier is used, it is usually as part of the NLP process. This study examines using classifiers directly for assessing short text answers to questions and as such contributes to the area of machine learning in intelligent tutoring systems. Four classifiers, namely, support vector machines, random forest, gradient boosting and an ensemble using all three classifiers are investigated for automated marking of short answers. The performance of the classifiers was compared to approaches using similarity indexes (Sørensen–Dice index, Tversky similarity measure, cosine similarity) and fuzzy ratios (fuzzy partial ration, fuzzy weighted ratio, token sort ratio) previously examined for this purpose. The approaches were evaluated on the Beetle and SciEntsBank datasets. The classifiers were found to outperform the approaches using the similarity indexes and the fuzzy ratios. The random forest produced the best results for the Beetle dataset and the ensemble for the SciEntsBank dataset. The performance of these classifiers was also found to be comparative to traditional NLP approaches applied to these datasets sets.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Study of Classifiers for Marking Short Answers in Intelligent Tutoring Systems

  • Nelishia Pillay,
  • Gisele Marais

摘要

Traditionally natural language processing (NLP) approaches are used for automated marking of short answers in intelligent tutoring systems. If a classifier is used, it is usually as part of the NLP process. This study examines using classifiers directly for assessing short text answers to questions and as such contributes to the area of machine learning in intelligent tutoring systems. Four classifiers, namely, support vector machines, random forest, gradient boosting and an ensemble using all three classifiers are investigated for automated marking of short answers. The performance of the classifiers was compared to approaches using similarity indexes (Sørensen–Dice index, Tversky similarity measure, cosine similarity) and fuzzy ratios (fuzzy partial ration, fuzzy weighted ratio, token sort ratio) previously examined for this purpose. The approaches were evaluated on the Beetle and SciEntsBank datasets. The classifiers were found to outperform the approaches using the similarity indexes and the fuzzy ratios. The random forest produced the best results for the Beetle dataset and the ensemble for the SciEntsBank dataset. The performance of these classifiers was also found to be comparative to traditional NLP approaches applied to these datasets sets.