Coronary computed tomography angiography (CCTA) plays a key role in the non-invasive evaluation of coronary artery disease (CAD) severity. Typically, clinicians assign CAD-Reporting and Data System (CAD-RADS) scores based on visual inspection to determine the degree of vascular stenosis. However, traditional CAD-RADS assessments rely heavily on subjective interpretation, resulting in a time-consuming diagnostic process. Although some studies use deep learning methods for automatic diagnosis, due to the challenges in assembling large-scale medical imaging datasets, applying deep learning for automatic classification remains difficult. To overcome these constraints, we propose an automated CAD-RADS scoring framework that integrates MaxViT with diffusion models. First, we collect 1,232 CCTA images from 153 patients and construct the Dif-CCTA dataset using synthetic image data generated by a MedDiffusion module which effectively addresses overfitting. Second, we propose a multi-angle spatial information fusion strategy during training to counteract spatial information loss commonly encountered in single-image learning approaches. Our framework achieves accuracies of 0.92 and 0.91 in case-level and image-level experiments, respectively. Results from ablation studies and comparative evaluations confirm the effectiveness of the framework, which surpasses other existing methods.