DETR-like [5, 6, 11, 13, 30] models are effective in segmentation tasks. However, these models always use many complex transformer blocks in the decoder. Therefore, these models may not be suitable for segmentation tasks that require high real-time performance. To address this problem, we propose a highly real-time model called DETR-SS. DETR-SS uses a new lightweight decoder to significantly reduce the model complexity and accelerate the execution speed. The new decoder adapts dual-channel attention to replace the original cross-attention to obtain a sufficient receptive field. In addition, an effective feature fusion module is used to further improve our model’s accuracy. It uses our proposed attention refinement module to improve prediction accuracy. Extensive experiments are performed on Cityscapes and CamVid datasets. Our model achieved a Mean IOU of 78.3% and 80.5% at 2048  \(\times \)  1024 and 960  \(\times \)  720 resolutions on the Cityscapes and CamVid datasets, respectively. We also achieved 86.9 FPS and 169.4 FPS on both datasets respectively. The proposed models not only achieve high segmentation accuracy but also meet the requirement of real-time performance. We hope that our proposed DETR-SS model will contribute to further research on future real-time segmentation tasks.