The demand for high-quality, large-scale labeled datasets has become critical to the success of machine learning (ML) models. Traditional crowdsourcing platforms, such as Amazon Mechanical Turk, have facilitated the labeling process by leveraging distributed participants to annotate data. However, ensuring data integrity and label quality in crowdsourced environments remains a persistent challenge, particularly in the presence of inconsistent or malicious inputs. This study introduces a blockchain-based solution designed to enhance the reliability and transparency of crowdsourced data labeling. By integrating blockchain technology with a user rating system, the study aims to provide a decentralized, immutable framework that improves the accuracy of labels through two consensus mechanisms. The first mechanism enforces agreement based on a predefined number of users, while the second uses a rating threshold to determine the final label. Through simulation and evaluation, the system is tested under varying proportions of truthful and non-truthful users to assess label accuracy, agreement rates, and user ratings. The results demonstrate that the proposed approach offers a scalable and reliable solution for crowdsourced data labeling, with potential applications in improving ML model performance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Blockchain Meets Crowdsourcing: A Consensus Mechanism for Robust Data Labeling

  • Marcel Pehlke,
  • Sophia Fedder,
  • Mike Witkowski,
  • Clemens Schmitt,
  • Marc Jansen

摘要

The demand for high-quality, large-scale labeled datasets has become critical to the success of machine learning (ML) models. Traditional crowdsourcing platforms, such as Amazon Mechanical Turk, have facilitated the labeling process by leveraging distributed participants to annotate data. However, ensuring data integrity and label quality in crowdsourced environments remains a persistent challenge, particularly in the presence of inconsistent or malicious inputs. This study introduces a blockchain-based solution designed to enhance the reliability and transparency of crowdsourced data labeling. By integrating blockchain technology with a user rating system, the study aims to provide a decentralized, immutable framework that improves the accuracy of labels through two consensus mechanisms. The first mechanism enforces agreement based on a predefined number of users, while the second uses a rating threshold to determine the final label. Through simulation and evaluation, the system is tested under varying proportions of truthful and non-truthful users to assess label accuracy, agreement rates, and user ratings. The results demonstrate that the proposed approach offers a scalable and reliable solution for crowdsourced data labeling, with potential applications in improving ML model performance.