The precise prediction of Pathological Complete Response (pCR) following Neoadjuvant Chemo-ImmunoTherapy (NCIT) in Head and Neck Squamous Cell Carcinoma (HNSCC) is crucial for optimizing therapeutic strategies and prognostic evaluation. Current methods exhibit limitations in simultaneously modeling multi-temporal treatment dynamics, multi-sequence magnetic resonance imaging (MRI) correlations, and multi-modal feature interactions. To address this challenge, we present a novel multi-modal representation and fusion framework, HARM \(^{\text {3}}\) -Fusion, which innovatively processes multi-temporal, multi-sequence MRI data and hierarchically fuses it with whole slide image (WSI) to enhance the accuracy of pCR prediction. Specifically, our method comprises three key modules: a multi-temporal MRI fusion module based on Loss-enhanced Dual-stream Convolutional Variational Auto-Encoder (LD-VAE), designed to decouple features from pre-treatment and post-treatment MRI scans; a multi-sequence MRI fusion module based on self-attention for integrating MRI features from T1 and T2 weighted sequences; and a multi-modal MRI-WSI fusion module based on cross-attention to fuse complementary information between MRI and WSI. To evaluate the efficacy of HARM \(^{\text {3}}\) -Fusion, we establish HNSCC-pCR, the first multi-modal dataset for HNSCC. HNSCC-pCR dataset comprises 407 patients, with each case including pre-treatment and post-treatment T1-weighted and T2-weighted MRI scans, WSI of pre- biopsy specimens, and pathologically confirmed surgical pCR. Based on this dataset, experimental results demonstrate that HARM \(^{\text {3}}\) -Fusion achieves superior performance for pCR prediction compared to other single-modal and multi-modal approaches.