Automated content analysis involves applying AI to insufficiently formalized areas of social studies and humanities. In 2023–2024, we researched the landscape of human values, represented in Russian social media writings, compiled a classifier, and labeled the collected data by 105 human values and their tonality. This paper discusses our development pipeline for generalized text markup structures, the challenges of machine learning to automatically identify human values in texts, own benchmark performance, the Human Values dataset in Russian, and evaluation procedures. The data set is used to create a RoBERTa-based model and a prompt-based model with Qwen 2.5 14B Instruct. These new models automatically categorize texts and focus on multi-annotator assessments. Benchmarks are created to compare the performance of these ML models with human annotators and a Mixture of Expects strategy. Both models demonstrate relatively high performance in detecting groups of human values, and sentiment classification does not significantly degrade it. The RoBERTa-based model exhibits a higher average performance (F1 0.753) in detecting parent classes in our hierarchical classifier compared to the Qwen-based model (F1 0.739); both indicate a good balance between precision and recall. In recognizing human values at child levels in the classifier RoBERTa (F1 0.274) performs significantly less effectively than the prompt-based strategy (F1 0.537). The dependence of the F1 score, obtained in the validation stage, on the number of annotated instances per class, used in training the RoBERTa-based model, demonstrates an almost linear relationship between the performance achieved and the number of annotated instances.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automated Detection of Human Values in Texts: ML Challenges and Performance Benchmarks

  • Olga Rink,
  • Archil Maysuradze,
  • Artem Fedorov,
  • Roman Ischenko,
  • Anna Korchagina,
  • Andrey Tabachenkov,
  • Ilya Tsybanov,
  • Konstantin Vorontsov

摘要

Automated content analysis involves applying AI to insufficiently formalized areas of social studies and humanities. In 2023–2024, we researched the landscape of human values, represented in Russian social media writings, compiled a classifier, and labeled the collected data by 105 human values and their tonality. This paper discusses our development pipeline for generalized text markup structures, the challenges of machine learning to automatically identify human values in texts, own benchmark performance, the Human Values dataset in Russian, and evaluation procedures. The data set is used to create a RoBERTa-based model and a prompt-based model with Qwen 2.5 14B Instruct. These new models automatically categorize texts and focus on multi-annotator assessments. Benchmarks are created to compare the performance of these ML models with human annotators and a Mixture of Expects strategy. Both models demonstrate relatively high performance in detecting groups of human values, and sentiment classification does not significantly degrade it. The RoBERTa-based model exhibits a higher average performance (F1 0.753) in detecting parent classes in our hierarchical classifier compared to the Qwen-based model (F1 0.739); both indicate a good balance between precision and recall. In recognizing human values at child levels in the classifier RoBERTa (F1 0.274) performs significantly less effectively than the prompt-based strategy (F1 0.537). The dependence of the F1 score, obtained in the validation stage, on the number of annotated instances per class, used in training the RoBERTa-based model, demonstrates an almost linear relationship between the performance achieved and the number of annotated instances.