CT scans are essential for identifying strokes, which are serious cerebrovascular conditions that result in major morbidity, death, and long-term disability. Accurate and efficient diagnosis is crucial for effective therapy. Although CT scans are frequently employed for the first diagnosis of strokes, the interpretation of these images by radiologists can be intricate, underscoring the want for automated diagnostic techniques utilizing Artificial Intelligence (AI). The objective of this research work is to integrate domain experience with artificial intelligence to develop a model for stroke diagnosis using CT scans. The integration of a Vision Transformer (ViT) with a Long Short Term Memory (LSTM) network in our system enables precise identification of stroke characteristics. The ViT algorithm extracts crucial characteristics from computed tomography (CT) pictures, while the Long Short-Term Memory (LSTM) model analyzes sequential data, collecting time-related connections that are crucial for comprehending patterns in the data. Our model successfully tackles the issue of class imbalance in CT image datasets, surpassing previous techniques in terms of accuracy and performance. After performing thorough preprocessing, training on the stroke CT dataset from Rajshahi Medical College Hospital in Bangladesh resulted in remarkable accuracies. The model attained peak accuracies of 73.8%, 91.61%, 93.5%, and 94.55% while employing the SGD, RMSProp, Adam, and AdamW optimizers, respectively. These results were verified using a thorough fivefold cross-validation process. Moreover, Explainable AI (XAI) methods were employed to improve the model’s comprehensibility, enabling doctors to grasp its decision-making process.