Lesion segmentation is an essential task in medical imaging, aiding in the diagnosis and assessment of pulmonary diseases. While multi-modal approaches combining text with images improve segmentation by offering complementary cues, existing multi-modal models often utilize only a single abstract text source, and do not fully exploit its hierarchical interactions with visual features. In this work, we propose a novel multi-modal segmentation framework, i.e., REMix that refines and mixes image and text representations throughout the hierarchical decoding process. By adaptively structuring textual information and enhancing visual representations, our method effectively aligns both high-level semantics and fine-grained details. Extensive experiments on the QaTa-COV19 and MosMedData+ datasets demonstrate that our approach achieves superior segmentation performance, outperforming existing uni-modal and multi-modal methods.