Thoracic Aortic Aneurysms (TAA) are serious cardiovascular conditions that frequently go unnoticed until that disease has grown to a critical stage. Therefore, early identification is crucial to prevent death from TAA. To automatically identify TAA from chest X-Rays, five deep learning architectures were evaluated: ResNet, DenseNet, Vision Transformer (ViT), Swin Transformer, and a Hybrid Vision Transformer with CNN feature extractor. Experimental results showed that transformer-based architectures outperformed conventional convolutional neural networks. The hybrid ViT+ CNN model was the most accurate, with 93% accuracy rate. Swin Transformer with 90%, ViT with 88%, DenseNet with 87%, and ResNet with 62% accuracy rate. The results suggest that advanced deep learning models, including transformer-based models, can securely and precisely identify TAA. AI-based diagnostic tools are therefore well-positioned to improve clinical outcomes and address significant healthcare challenges. Although similar transformer based medical imaging methods have been proposed by directly applying Vision Transformers to the raw images, the proposed hybrid architecture of ViT+ CNN combines CNN-based local feature extraction with the transformer based global context in a way that is specifically suited to the detection of abnormalities in the thoracic aorta on chest X-rays.