Alzheimer’s disease (AD) is an incurable, progressive neurodegenerative disease, and its early diagnosis is essential. Previous studies have demonstrated the superiority of utilizing multi-site data to train models for diagnosing AD. However, existing research models mostly use dimensionally reduced image data for model training, which loses the complete spatial information of the image. At the same time, the data from different sites are heterogeneous, and the joint training models often perform poorly when facing data from new sites. Traditional domain adaptation methods require centralized data for training, which results in the leakage of medical privacy. Our study proposes a multi-site federated learning model (FedSADA), which uses a 3D Vision Transformer as the basic framework to learn the spatial information of the complete image fully. We also added a self-attention domain adaptive loss function and a local maximum mean difference loss function to perform domain adaptation on multi-site data. At the same time, a federated learning framework is adopted to protect medical data fully. We conducted experiments on public datasets, demonstrating that our method can effectively improve model generalization and accuracy on new data sites.