Amyloid- \(\beta \) (A \(\beta \) ) accumulation patterns (i.e., overall load and spatial distribution) have become important diagnostic biomarkers for Alzheimer’s disease (AD). However, A \(\beta \) accumulation is often classified in a binary manner through A \(\beta \) -PET, which oversimplifies the complexity of A \(\beta \) accumulation patterns. Thus, further stratification of A \(\beta \) accumulation patterns is essential to unravel the heterogeneity of A \(\beta \) accumulation across individuals and advance our understanding of AD. However, existing stratification methods often rely solely on A \(\beta \) -PET, which limits the spatial localization of A \(\beta \) accumulation patterns due to the lack of detailed anatomical information. To overcome this limitation, we propose a multi-modal latent space clustering method that leverages residual diffusion models to better uncover A \(\beta \) accumulation patterns during the process of synthesizing A \(\beta \) -PET from MRI. Specifically, to enhance the spatial localization of A \(\beta \) accumulation patterns, we construct a multi-modal latent space that integrates both A \(\beta \) -PET and MRI data by employing a residual diffusion model to synthesize A \(\beta \) -PET from MRI. Additionally, to improve the efficiency of latent space refinement, we encode mixtures of MRI and PET data at different stages of the diffusion process by progressively adding and removing residual differences between MRI and PET. Finally, we perform k-means clustering on the constructed latent space to uncover A \(\beta \) accumulation patterns. Experiments demonstrate that our method not only effectively identifies meaningful subtypes that align well with established subtypes, but also reveals intermediate subtypes that go beyond the scope of these established subtypes, offering new insights into A \(\beta \) accumulation patterns.