Label Engineering Methods for ML Systems

The goal of label engineering is to generate labels that accurately reflect the intended meaning or purpose of the machine learning model, while also being useful and relevant for the downstream artificial intelligent systems. These range from generating labels for computer vision use cases, natural language tasks, language models, and other custom enterprise models. Label engineering is becoming a hot topic when it comes to training intelligent systems with initial ground truth data and iterating on existing models for improvement, for large-scale business use cases. The processes for generating ground truth data through labeling and natural language annotations can be very cumbersome, because very few standardization approaches have been researched on. In this paper, we review the existing literature and present the challenges in label engineering. We also propose methodologies that can help generate trustworthy and reliable labels through well-governed and regulated processes and pipelines. It is important to understand the benefits and drawbacks of different approaches and choose which best suits our use cases. After understanding these challenges and choosing what approach to take during the architecture phase, it becomes important to have explainability for the model. In case of unexpected outcomes, the model can be traced back to the set of labels that led to the model training in a direction leading to the problematic classification. We conclude the paper with initial experiments and recommendations.