From Bias to Fairness: The Role of Domain-Specific Knowledge and Efficient Fine-Tuning

In the field of Natural Language Processing (NLP), Large Language Models (LLMs) demonstrate proficiency in complex tasks and are widely used in many real-world applications. However, they are influenced by biases inherent in their pre-trained data, a phenomenon known as “prior knowledge”. To address this challenge, this study focuses on leveraging domain-specific knowledge about sensitive attributes for activating and reorganizing the prior knowledge in LLMs, thereby aiming to diminish model bias. To be specific, our methodology involves the construction of domain knowledge around sensitive attributes via Counterfactual Data Augmentation (CDA) and the efficiently fine-tuning of model parameters through Low-Rank Adaptation (LoRA) techniques. Furthermore, this research assesses the influence of training data volume and training data content on the model’s bias reduction capabilities. On several publicly available benchmark datasets, the experimental results suggest that using the optimal training volumes and choosing training content that is related to the model’s learned knowledge can significantly mitigate bias while preserving language modeling performance. This study offers novel insights and strategies for addressing fairness in LLMs.