A decoder-free feature aggregation network for medical image segmentation

Encoder-decoder structured networks have recently become dominant in medical image segmentation, primarily due to their straightforward architecture. However, their extensive use of up-sampling operations, aimed at expanding network parameters, often leads to feature information loss. To address this issue, we propose a novel, decoder-free architecture for medical image segmentation, termed Feature Aggregation Network (FAN-Net). This approach eliminates both the pooling operation and the decoder phase, directly addressing the loss of feature information. During the encoding phase, we replace the traditional pooling operation with the patch merge attention (PMA) module, which systematically reduces feature map dimensionality. Despite omitting the decoding structure, the necessity to restore feature map resolution remains. To this end, we introduce the multi-path feature aggregation (MFA) module, which reinstates feature map dimensionality. Additionally, we propose an adaptive attention feature (AAF) extraction module, engineered to enhance semantic representation distinctiveness. We conduct extensive experiments on popular semantic segmentation benchmarks including IXI dataset, DRIVE, and CHASEDB. The experimental results demonstrate the effectiveness of the proposed FAN-Net, which is capable of achieving better segmentation performance than other state-of-the-art approaches. In addition, this paper also shows that medical image segmentation can be performed without necessitating the deployment of the decoder stage.