Accurate tumor delineation in radiotherapy requires synergistic analysis of multi-modal data. However, current automated methods are predominantly limited to single imaging modalities. We introduce a multi-modal segmentation framework that integrates 3D CT and MRI volumes with clinical text descriptions. Our architecture processes CT and MRI data through shared encoders with modality-specific normalization. A hierarchical cross-attention decoder enables multi-scale fusion of radiometric features and semantic text embeddings. Additionally, a text-guided boundary refinement module uses tumor location and quantity descriptors to accurately segment tumor regions. Evaluated on two public datasets, LiTS (CT+Text) and ATLAS (MRI+Text), our method achieved superior performance in tumor segmentation, with up to 16% improvement in mean Dice scores over existing state-of-the-art methods. Ablation studies confirmed the complementary benefits of image-text integration. The results demonstrate that our multi-modal learning approach enhances segmentation accuracy, particularly for small tumor regions.