Humans have traditionally been an integral part of artificial intelligence systems as a means of generating labeled training data [5, 26, 42, 48, 54, 64, 73]. Such a paradigm has been proven to be effective in supervised learning tasks such as image classification [13], speech recognition [20], autonomous driving [65], social media mining [83], and virtual reality [61]. However, it also suffers from two key limitations. First, some applications (e.g., disaster response and damage assessment, online truth discovery) may require a large amount of training data to achieve reasonable performance, which could be impractical due to the labor cost [22, 37]. Second, the AI models are often black-box systems and it is difficult to diagnose in the event of failure or unsatisfactory performance. To address these limitations, a few human-AI hybrid frameworks have been developed in recent years. For example, Holzinger et al. proposed the notion of interactive human machine learning (“iML”), where humans directly interact with AI by identifying useful features that could be incorporated into the AI algorithms [23]. Branson et al. invented a human-in-the-loop visual recognition system to accurately classify the objects in the picture based on the descriptions of the picture from humans [7]. More recently, researchers have been interested in diagnosing the black-box AI algorithms to provide accountability. For example, Nushi et al. developed an accountable human-AI system that leverages workers on Amazon Mechanical Turk (AMT) to identify the limitations of the AI algorithms [49] and provide suggestions to improve them. However, the above solutions largely ignored the innate limitations of the AI algorithms that cannot be simply improved by retraining the model with more data. Human-AI interaction is a trending research area that aims at harnessing the power of human intelligence and AI to optimize the human-AI systems and improve human experience with AI [11, 40, 52, 56, 71]. Such a paradigm has been applied in many domains, including image classification [50, 60], natural language translation [6, 86], medical diagnosis [14, 39], and autonomous driving [74, 84]. More recently, a few human-AI interaction systems have been developed to explore the human intelligence of crowd workers through interactive crowdsourcing tasks [41, 46]. For example, Nguyen et al. designed a human-AI interactive news article fact-checking algorithm that checks the truthfulness of textual news and claims by assigning crowd workers to correct identification errors by AI models [46]. Mandel et al. developed a game-based crowdsourcing interface to incorporate a crowd of non-AI experts to reason the dynamics of AI misbehavior and improve AI performance in online advertisement recommendations [41]. However, these solutions either assume the human workers have sufficient domain knowledge or require them to be well-trained on domain-specific crowdsourcing tasks. Such approaches often suffer from noisy crowdsourcing results since the ordinary crowd workers usually do not have the essential domain knowledge for the domain-specific tasks or are not interested in those training tasks [29, 76]. Future works in the direction of human-AI collaboration systems are expected to address some of these limitations.

Further Readings

章节
发表日期：

2025年

查看全文

Dong Wang,
Lanyu Shang,
Yang Zhang

摘要

Humans have traditionally been an integral part of artificial intelligence systems as a means of generating labeled training data [5, 26, 42, 48, 54, 64, 73]. Such a paradigm has been proven to be effective in supervised learning tasks such as image classification [13], speech recognition [20], autonomous driving [65], social media mining [83], and virtual reality [61]. However, it also suffers from two key limitations. First, some applications (e.g., disaster response and damage assessment, online truth discovery) may require a large amount of training data to achieve reasonable performance, which could be impractical due to the labor cost [22, 37]. Second, the AI models are often black-box systems and it is difficult to diagnose in the event of failure or unsatisfactory performance. To address these limitations, a few human-AI hybrid frameworks have been developed in recent years. For example, Holzinger et al. proposed the notion of interactive human machine learning (“iML”), where humans directly interact with AI by identifying useful features that could be incorporated into the AI algorithms [23]. Branson et al. invented a human-in-the-loop visual recognition system to accurately classify the objects in the picture based on the descriptions of the picture from humans [7]. More recently, researchers have been interested in diagnosing the black-box AI algorithms to provide accountability. For example, Nushi et al. developed an accountable human-AI system that leverages workers on Amazon Mechanical Turk (AMT) to identify the limitations of the AI algorithms [49] and provide suggestions to improve them. However, the above solutions largely ignored the innate limitations of the AI algorithms that cannot be simply improved by retraining the model with more data. Human-AI interaction is a trending research area that aims at harnessing the power of human intelligence and AI to optimize the human-AI systems and improve human experience with AI [11, 40, 52, 56, 71]. Such a paradigm has been applied in many domains, including image classification [50, 60], natural language translation [6, 86], medical diagnosis [14, 39], and autonomous driving [74, 84]. More recently, a few human-AI interaction systems have been developed to explore the human intelligence of crowd workers through interactive crowdsourcing tasks [41, 46]. For example, Nguyen et al. designed a human-AI interactive news article fact-checking algorithm that checks the truthfulness of textual news and claims by assigning crowd workers to correct identification errors by AI models [46]. Mandel et al. developed a game-based crowdsourcing interface to incorporate a crowd of non-AI experts to reason the dynamics of AI misbehavior and improve AI performance in online advertisement recommendations [41]. However, these solutions either assume the human workers have sufficient domain knowledge or require them to be well-trained on domain-specific crowdsourcing tasks. Such approaches often suffer from noisy crowdsourcing results since the ordinary crowd workers usually do not have the essential domain knowledge for the domain-specific tasks or are not interested in those training tasks [29, 76]. Future works in the direction of human-AI collaboration systems are expected to address some of these limitations.