Automated Hierarchical Conflict Reduction for Crowdsourced Annotation Tasks Using Belief Functions

A typical crowdsourcing task is concept labeling, where participants annotate e.g. images using a list of predefined concepts. Recent popular campaigns for environmental bird monitoring even use hierarchies of concepts (taxonomies of species) to obtain the most precise labeling of bird images. But in most applications, volunteer opinions are isolated from each other, and decision is taken upon majority voting. In this work we propose a new iterative labeling process where participants express their opinions together, on ascending levels of the taxonomy. Level changes are performed to minimize opinion conflict, according to the belief function theory. This complex task is orchestrated by a finite-state automaton driven by conflict measures.