Action Quality Assessment (AQA) typically refers to the use of machine learning algorithm to evaluate the quality of executed actions, rather than relying on manual scoring. It usually believes that due to the subjective preferences of individuals, AQA scoring is more objective and more consistent. However, machines can be biased as well, as data-driven algorithms often inherit biases from their training data, leading to unfair outcomes. In order to eliminate bias from features, we propose a Causal Debiasing Network (CDN) method in this paper. By implementing causal inference to eradicate bias within individualized characteristics, this approach enhances the fairness of AQA, ensuring assessments remain unbiased and centered on action-related features. Experimental results on two widely used public datasets (MTL-AQA and AQA-7), demonstrate exemplary performance of our CDN, and especially in terms of scoring consistency, marked by a notable decrease in standard deviation, with the scoring accuracy now exhibiting a 1.42% improvement and variances now exhibiting a 1.43% decrease.