Survival prediction, which estimates the time from a clinical event to a specific outcome using multi-modal is crucial for cancer prognosis. Traditional statistical models, including the Cox model and Kaplan–Meier curve, suffer from strong assumptions of linearity and proportionality, limiting their ability to handle high-dimensional or non-linear features. Recently, deep learning approaches have shown promise in addressing these limitations by integrating heterogeneous data sources, particularly whole slide images (WSIs) and genomic profiles. However, existing multi-modal fusion strategies often neglect inter-modal interactions or are hindered by feature redundancy in WSIs due to their gigapixel scale and intra-tissue similarity. To address these challenges, we propose a novel Cross-Attention Transformer with Textual Prompt Learning framework for survival prediction. Our method leverages semantic-guided textual prompts to refine visual features from WSIs and introduces a cross-modal cross-attention module to capture intrinsic correlations and facilitate complementary information exchange between modalities. Experimental results on five TCGA datasets demonstrate that our approach outperforms state-of-the-art methods. Our code is publicly available at https://github.com/Peiweiwang0/CAT-with-TPL for reproducibility and further research.