Influence of AI’s Uncertainty in the Dawid-Skene Aggregation for Human-AI Crowdsourcing

The power and expressiveness of AIs are rapidly increasing, and now AIs have the ability to complete tasks in crowdsourcing as if they were human crowd workers. Therefore, the development of methods to effectively aggregate the results of tasks performed by AIs and humans is becoming a critical problem. In this study, we revisit the Dawid-Skene model that has been used to aggregate human votes to obtain better results in classification problems. Most of the state-of-the-art AI classifiers predict the class probabilities as their output. Considering the probabilities represent their uncertainty, utilizing them in Dawid-Skene aggregation may provide higher-quality annotations. To this end, we introduce a variation of the Dawid-Skene model to directly use the probabilities without discarding them and conduct experiments with two real-world datasets of different domains. Experimental results show that the Dawid-Skene model with probabilities improves the overall accuracy. Moreover, a detailed analysis shows that the aggregation results were improved for classification tasks with high uncertainty.