Alzheimer’s disease (AD) is a progressive neurodegenerative disorder characterized by a gradual decline in cognitive function. The brain structure of AD patients progressively atrophies over time. Longitudinal data can capture the changes in imaging over time, providing important information for early detection and diagnosis of the disease, which is not achievable with images from a single time point. We propose a novel longitudinal AD diagnosis model that leverages deformation field data obtained by registering images from adjacent time points. Each pixel in the deformation field represents the displacement vector of the corresponding pixel between two scans, reflecting information about brain tissue atrophy. By effectively extracting both temporal and spatial features from the longitudinal data, our model enhances early AD diagnosis. The input data for the model consists of processed sMRI images and deformation fields calculated between each pair of time points in the longitudinal sMRI data. The model is composed of three modules: Convolutional Long Short-Term Memory (CLSTM), Gate Fusion Long Short-Term Memory (GFLSTM), and an attention module. CLSTM extracts temporal features while preserving spatial features by maintaining the 3D shape of the features, GFLSTM achieves spatiotemporal feature fusion within a cascaded network, and the attention module further enhances feature representation. Our proposed method was evaluated on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database, and experiments demonstrate that it outperforms several state-of-the-art methods.