Dense object detection has been popular for years with the success of the multi-level learning framework. By delivering the learning of objects into a multi-level feature pyramid, such a divide-and-conquer solution eases the optimization difficulty. However, a commonly neglected problem is that the shallow levels take tons of computation due to their high resolutions of the feature maps, heavily slowing down the inference speed. To address this issue, we explore multi-level head network design by investigating performance sensitivity. The outcome is SlimHead, a simple, efficient, and generalizable head network, which further unleashes the potential of multi-level learning for dense object detectors. It operates in two stages: Slim and Fat, initially plugging interpolator before the head network functions to “slim” the feature pyramid, and then recovering the features to original solution space by “fatting” the feature pyramid. Thanks to its flexibility, operations with higher computational complexity can be easily integrated to benefit accuracy without loss of inference efficiency. We also extend our SlimHead to multiple high-level vision tasks such as rotated object detection, pedestrian detection, and instance segmentation. Extensive experiments on PASCAL VOC, MS COCO, DOTA, and CrowdHuman demonstrate the broad applicability and the high practical value of our method.