Scholarlink

<p>Student evaluations of teaching (SET) act as the primary means to gauge instructor effectiveness. Likewise, SETs provide the primary qualitative feedback to instructors via student comments. However, mostly students with strong feelings tend to write comments. Among the most recallable are toxic comments: comments that are unhelpful/hurtful in harassment, outrage, or personal attacks. These comments demoralize instructors, while unduly influencing administrator hiring/firing decisions. To date, most universities do not systematically identify and quarantine toxic comments. Therefore, we ask, how well can automated machine learning methods systematically classify toxic comments? We created a 20-item codebook to train human coding of toxicity for the purpose of labeling SET comments as toxic across three universities. These data are used to rate competing toxicity classifiers. We find that our human coding reaches moderate to strong intercoder reliability, a necessary condition for classifying data. Additionally, we find that of the competing Machine Learning (ML) models, the pre-trained Perspective model minimizes false positives and maximizes true positives in a manner that far exceeds traditional first stage classifiers. Therefore, ML toxicity classification can be used to efficiently identify and eliminate toxic comments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Automating Identification of Toxic Comments in Student Evaluations of Teaching

  • 期刊论文

  • 发表日期：

    2025年6月12日

查看全文
  • Koen Suzelis,
  • Gabriel Mott,
  • John Curiel

摘要

Student evaluations of teaching (SET) act as the primary means to gauge instructor effectiveness. Likewise, SETs provide the primary qualitative feedback to instructors via student comments. However, mostly students with strong feelings tend to write comments. Among the most recallable are toxic comments: comments that are unhelpful/hurtful in harassment, outrage, or personal attacks. These comments demoralize instructors, while unduly influencing administrator hiring/firing decisions. To date, most universities do not systematically identify and quarantine toxic comments. Therefore, we ask, how well can automated machine learning methods systematically classify toxic comments? We created a 20-item codebook to train human coding of toxicity for the purpose of labeling SET comments as toxic across three universities. These data are used to rate competing toxicity classifiers. We find that our human coding reaches moderate to strong intercoder reliability, a necessary condition for classifying data. Additionally, we find that of the competing Machine Learning (ML) models, the pre-trained Perspective model minimizes false positives and maximizes true positives in a manner that far exceeds traditional first stage classifiers. Therefore, ML toxicity classification can be used to efficiently identify and eliminate toxic comments.