Assessment is one of the key stages of the decision-making process. In the con-text of the widespread digitalization of society, software systems that automate certain aspects of human activity that require the assessment issues are increasingly being used. The effectiveness of the practical application of such systems is directly related to the extent to which users can trust the results obtained in such systems. A method for analyzing the trustworthiness of assessments based on the principle of trust transitivity is proposed. The method interprets the principle as the possibility of transferring trust from the expert to the system, considering both as independent assessors. The ability to transfer trust in the evaluation of an assessor, whose trustworthiness the trustor does not doubt, to the evaluation of an assessor outside the trust relation is determined by the degree of consistency between the pair of assessments obtained from the assessors. Setting a threshold for the consistency measure sufficient for trust transfer is the responsibility of the trustor. The method was tested on a system for automated assessment of the functional state of a software user. To measure the consistency between the assessments obtained from the experts and from the system, a linear consistency measure was used. The testing results showed that proposed method is applicable for analyzing the trustworthiness of assessments obtained in software systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Trust in Automation: Evaluation of Trustworthiness of Automated Assessment

  • Aleksandr Davtian,
  • Olga Shabalina,
  • Olga Berestneva,
  • Irina Molodtsova,
  • Elena Khvastunova,
  • Anna Andropova

摘要

Assessment is one of the key stages of the decision-making process. In the con-text of the widespread digitalization of society, software systems that automate certain aspects of human activity that require the assessment issues are increasingly being used. The effectiveness of the practical application of such systems is directly related to the extent to which users can trust the results obtained in such systems. A method for analyzing the trustworthiness of assessments based on the principle of trust transitivity is proposed. The method interprets the principle as the possibility of transferring trust from the expert to the system, considering both as independent assessors. The ability to transfer trust in the evaluation of an assessor, whose trustworthiness the trustor does not doubt, to the evaluation of an assessor outside the trust relation is determined by the degree of consistency between the pair of assessments obtained from the assessors. Setting a threshold for the consistency measure sufficient for trust transfer is the responsibility of the trustor. The method was tested on a system for automated assessment of the functional state of a software user. To measure the consistency between the assessments obtained from the experts and from the system, a linear consistency measure was used. The testing results showed that proposed method is applicable for analyzing the trustworthiness of assessments obtained in software systems.