Computer-aided diagnosis (CAD) systems demand lightweight and efficient models that maintain high accuracy while minimizing computational overhead for practical use. To this end, we propose a lightweight Diffusion-based Fourier Neural Operator-aided U-Net model (DFU-Net) for medical image segmentation. The model performs exceptionally well on three benchmark segmentation datasets, namely TNBC, CPM17 and MoNuSeg, achieving state-of-the-art results with Dice scores of 0.9617, 0.9592, and 0.9096 and IoU scores of 0.9262, 0.9216, and 0.8350, respectively. Despite its high accuracy, the model remains highly efficient, with only 82.49K parameters. To evaluate its real-time feasibility, we tested the model on three edge devices: Raspberry Pi 4b, Raspberry Pi 5, and Nvidia Jetson Nano. Among these, Raspberry Pi 5 delivers the best performance, processing 128 \(\,\times \,\) 128 images in just 0.37 s, demonstrating the model’s potential for real-time, edge-based CAD applications. Code is available at: https://github.com/asfakali/DFU-Net .