Colorectal cancer (CRC) can be prevented with early detection and removal of the colorectal polyps. The accurate removal of polyps depends on the efficient segmentation of the polyps. In a power-constrained environment such as embedded devices, less complex deep learning models are needed to perform segmentation efficiently. This work proposed a lightweight segmentation model to be deployed on FPGA. The proposed architecture uses depthwise separable convolution efficiently, making the architecture perform well with lower computational complexity. The issue of polyp size variation can be successfully overcome with feature fusion within the encoder section of the architecture. The proposed architecture was successfully evaluated on CVC-CliniDB and Kvasir-SEG datasets with an mIoU of 0.9365 and 0.8501, respectively. It delivers a throughput of 44.82 FPS with 2.5011 energy efficiency when deployed on FPGA ZCU104. The proposed model outperformed other considered state-of-the-art (SOTA) models in terms of performance.