Dynamic Labeling: A Control System for Labeling Styles in Image Annotation Tasks

Labeling style affects labeling efficiency and quality in image annotation tasks. For example, a “label quickly” style can increase labeling efficiency when the data are easy, and a “label carefully” style can increase label quality when the data are difficult. However, the selection of an appropriate labeling style is difficult as different annotators have different experiences and domain knowledge, affecting their subjective feelings of data difficulties (for example, User 1 feels Data A to be easy, while User 2 feels it difficult). In this paper, we propose “Dynamic Labeling” as a control system for labeling styles used in image-labeling tasks. Our control system analyzes the labeling behaviors of annotators (i.e., label selection time) and dynamically assigns an appropriate labeling style (label quickly or label carefully). We conducted a user study to compare a conventional “non-dynamic” and the proposed “dynamic” labeling approaches for an image-labeling task. The results suggest that Dynamic Labeling increased the label quality and labeling efficiency.