Object detection models trained on natural data often face challenges with scale imbalance, where objects of different sizes provide varying levels of useful features. Larger objects typically contain richer and more detailed information, while smaller objects often lack sufficient distinctive features and are more prone to being overwhelmed by background noise. This imbalance can push models into suboptimal learning states. While the model effectively locates meaningful features for larger objects, it may fail to locate noisy features for smaller objects. To address this issue, we proposed a scale margin loss motivated by reducing models’ overfitting degree for small objects. By adding a set of regularization terms, our scale margin loss can better locate noisy features for small objects, making the model’s overall generalization ability closer to the optimal. We tested our method with the state-of-the-art object detection models on two benchmarking datasets. The experiments demonstrate the effectiveness of our methods with 1.42%+ Average Precision on MS COCO and 3.37%+ Small Object Average Precision on VOC. Codes are available at https://github.com/Andisyc/ScaleMargin.