Survival analysis is paramount for cancer patients as it offers crucial prognostic insights for treatment planning. The performance of existing survival analysis methods is mainly limited by two factors: 1) inefficient extraction of features from multi-modal medical data, e.g., MR images and clinical diagnostic descriptions; and 2) inadequate focus on disease-relevant regions, e.g., primary tumor. To deal with these challenges, in this study, we propose a rectal cancer survival analysis model, dubbed as SurRecNet, which effectively fuse MR images and diagnostic descriptions and takes advantage of multi-task learning. Specifically, we introduce a cross-modality alignment module, aiming to precisely align diagnostic descriptions with MR images at a granular level and facilitate accurate survival analysis. Furthermore, SurRecNet simultaneously predicts tumor masks, relapse states, and survival outcomes by leveraging multi-task learning strategy, imitating the diagnostic process of radiologists to enhance prediction performance. Experimental results on a real clinical rectal multi-modal dataset demonstrate that our SurRecNet significantly outperforms the state-of-the-art methods.