Scene text segmentation is crucial for applications such as document analysis and text removal. however, existing methods face challenges due to their reliance on costly pixel-level annotations and limited generalizability. Although deep learning models and the Segment Anything Model (SAM) have advanced segmentation technology, their direct application to text segmentation suffers from domain gaps. To address this issue, we propose CHSAM, a SAM-based framework with a Conv2d-Adapter (an adapter with 2D convolution) and hierarchical decoding. First, the Conv2d-Adapter fine-tunes SAM’s image encoder using lightweight convolutional layers, enhancing local text feature extraction while significantly reducing the number of trainable parameters. Second, through convolutional feature augmentation and mask feature augmentation, a two-stage decoder extracts and enhances implicit sparse embeddings from image embeddings to refine text features and boundary granularity. Experiments on benchmark datasets demonstrate that CHSAM achieves superior segmentation performance, significantly outperforming the baseline SAM in terms of robustness to deformed text and unseen scenarios. Additionally, CHSAM supports efficient end-to-end inference without requiring complex post-processing steps.