<p>In the field of data-driven deep learning, the dataset quality is one of the essential factors that determine the neural network (NN) accuracy. During the sampling process, the sample may contain many task-irrelevant factors. However, there is a lack of study on the influence of task-irrelevant factors in the dataset on the NN accuracy. This paper analyzes the influence of task-irrelevant factors through mathematical derivation, and finds that regularization cannot solve the problem of task-irrelevant factors affecting the NN inference performance. To reduce the influence of task-irrelevant factors, this paper proposes an idea, making the mathematical expectation of task-irrelevant features in the training set to be zero. Furthermore, this paper proposes a method that can reduce the influence of the background factor on the image classification model. To verify the effectiveness of this method, a series of controlled experiments are conducted on the image classification and object detection task. Experimental results show that this method not only effectively improves the accuracy of the classification model, but also improves the precision of the object detection model. The code related with this paper can be download from <a href="https://github.com/CaedmonLY/BackgroundFactor/tree/master">https://github.com/CaedmonLY/BackgroundFactor/tree/master</a>.</p>