SafeText: A Unified Approach for Detecting and Mitigating Toxicity and Bias in Textual Data
摘要
This paper introduces an innovative approach to AI-powered content moderation, in the context of any textual content; either human-written or AI/Large Language Model generated. The first and foremost in content moderation is detection of bias and toxic language in the content. Once the biased or toxic sentences are identified, we need to debias or detoxify the sentences by rewording or rephrasing, keeping the contextual meaning of the content intact. To achieve this, we fine-tuned an mT5-small model for detection and a T5-base model for mitigation. Our training process incorporated a broad range of datasets, including open-source data and synthetic data generated using the advanced GPT-4 model. A pivotal element in our methodology is the utilization of synthetic data generation, significantly augmenting the diversity and realism of the training dataset. Contrastive learning is employed to further enhance the model’s capability to discern subtle nuances in content. Noteworthy is our detection model, which not only identifies toxicity and bias but also offers reasoning, thereby introducing a layer of explainability to the content moderation process. Our results illustrate that our comprehensive approach, integrating sophisticated techniques and diverse data sources, surpasses existing solutions. This achievement provides scalable and effective tools for advancing digital safety and fostering inclusivity.