Only Classification Head Is Sufficient for Medical Image Segmentation

Medical image segmentation is a pivotal research domain that has garnered widespread attention in contemporary medical diagnostics. In pursuit of enhancing network efficacy, researchers have taken great efforts to develop various well-designed decoders. Unfortunately, due to the limited medical training data, the issues of underfitting and overfitting frequently arise. To this end, we undertake plentiful experiments to decouple the encoder and decoder components, and obtain a critical finding that excessively complex decoders impede the encoder’s potentiality of feature extraction. Inspired by some remarkable image generation work, we devise a straightforward segmentation network, which incorporates a pre-trained encoder backbone network and a pixel classification head. Our network not only ensures adequate feature decoding ability but also maximizes feature representation capability of the backbone. Experimental results on four datasets of three tasks show the outstanding performance against the state-of-the-art methods. The source code will be publicly available at https://github.com/wei-hongbin/CHNet