Accurate segmentation of the retinal vessels is vital for the early diagnosis of ophthalmic conditions such as glaucoma and diabetic retinopathy. Existing deep learning approaches primarily focus on decoder optimization, often underutilizing the rich feature representations encoded during the early stages of processing. In this paper, we propose GeGLUNet, a novel attention-guided encoder-decoder framework that integrates Gated Gaussian Error Linear Units for enhanced non-linear representation and introduces attention gates in skip connections to enable semantically refined feature fusion. To further improve segmentation fidelity, we design a hybrid loss function that combines binary cross-entropy, Dice, and Jaccard terms with an edge-aware penalty and a novel Latent Patch-wise Contrastive Loss (LPCL) for robust feature discrimination. Our approach is evaluated on four public datasets, where it consistently achieves state-of-the-art results across multiple metrics, including Dice, mIoU, and HD95. Extensive ablation studies confirm the individual contributions of GeGLU activations, attention modules, and each component of the loss formulation. GeGLUNet demonstrates strong generalization and structural accuracy, offering a scalable and precise solution for segmentation of the retinal vessel in clinical imaging.