Significant progress has been made in AI-based prediction of therapeutic response to neoadjuvant chemotherapy (NAC) in breast cancer. However, current studies primarily rely on data from a single time point, neglecting the dynamic changes in tumor characteristics during treatment. To address this limitation, we propose a novel Dynamic Temporal Feature Difference Fusion (DTFDF) framework, which integrates image features from multiple time points throughout the treatment process to predict therapy response more precisely. Based on tumor spatial features, we design an innovative DTFDF strategy and introduce a treatment response-based triplet contrastive loss function to facilitate the learning of longitudinal tumor changes and enhance feature representation. Additionally, we incorporate biomarker prediction as an auxiliary task and introduce a feature decoupling-based multi-task learning module. This module generates feature representations for different tasks by accounting for both shared and task-specific information, improving response prediction. Experiments with data from 786 patients in the I-SPY 2 trial dataset demonstrate that our method achieves the highest AUC of 0.835 in predicting radiation therapy response, outperforming state-of-the-art (SOTA) approaches on longitudinal dynamic contrast-enhanced MRI data. Our source code is available at https://github.com/AlexNmSED/DTFDF .