The improvement in accuracy of object detection networks has been progressive in facilitating real vision applications. However, the increase in accuracy comes with the cost of increasing memory footprint and computations and it has presented ever increasing challenges for deployment of updated object detection networks. To address these challenges, we propose a structured pruning method called Fine-grained Regularization with Global Initialization (FRGI). Considering the varying impact of filters on the subsequent layers that process the same L1-norm but different means, FRGI introduces a mean-aware sparsity term during the global sparsification to promote near-equal means of the filters, then initializes the global pruned filters by prioritizing them based on the L1-norm. The expressive power on the pruned structures selected in the initialization is transferred with minimum loss in accuracy by applying fine-grained regularization. Moreover, for the residual blocks commonly found in object detection networks, FRGI averages the L1-norm of related filters. We show through extensive validations on the MS-COCO dataset that FRGI leads to more efficient object detection models for all sizes of object detection networks.