Automatic identification of esophageal lesions is crucial for early diagnosis of esophageal cancer. Esophageal endoscopy is a key diagnostic method for detecting esophageal cancer. However, there are challenges in accurately identifying esophageal cancer due to variations in brightness, image quality, and lesion patterns in esophageal endoscopic images. To address these challenges, we propose the ESO-PVT framework for esophageal lesion segmentation. This network adopts the fundamental architecture of Polyp-PVT and introduces a novel color space fusion attention called CSF to effectively mitigate inconsistencies in the color space of esophagoscopy images. The CSF attention mechanism extracts the LAB color space from the input RGB image to mitigate the adverse effects of inconsistent color and sampling equipment on tumor segmentation by integrating the two color spaces. Experimental results on a collected clinical esophageal image dataset show that the proposed method achieves the best results in terms of segmentation accuracy compared with mainstream segmentation methods.