Real-time semantic segmentation attempts to rapidly and accurately classify each pixel in videos or images, facilitating instantaneous comprehension of scene content. Despite significant advancements in this field, it remains a challenge to balance segmentation accuracy and inference speed with limited computational resources. To address this problem, we propose the attention-guided dual-branch network (AGDBNet). Specifically, a lightweight dual-branch network is proposed for the separate extraction of detail features and context features. Additionally, a hybrid pooling attention module (HPAM) based on a novel pooling strategy is designed to improve the context branch’s ability to extract context features. Furthermore, a dual attention fusion module (DAFM) is designed to combine the complementary features extracted from both branches. Finally, we evaluate AGDBNet through qualitative and quantitative experiments. Our method obtains 76.7% mIoU with 139.2 FPS and 80.1% mIoU with 150.8 FPS on Cityscapes and CamVid, respectively. These outcomes indicate that our method provides an excellent compromise among precision and speed.