Military vehicle classification in real-world battlefield scenarios does not always yield optimal performance using only visual features. This is primarily due to the presence of various natural and anthropogenic stressors, such as camouflage, noise, distortion, coloration and defilade, which often severely degrade the images of target objects. Traditional visual classifiers are inadequate in such situations, as they often require copious training data spanning the entire spectrum of image degradations, which is not always feasible. Consequently, achieving high classification accuracy in real-world battlefield scenarios is a challenging task. To address this challenge, we propose a vision transformer (ViT)-based classifier termed ViTAtr that fuses ViT-based visual features and attribute-based features using a late fusion strategy for military vehicle classification in battlefield-stressed scenarios. ViTAtr aims to improve classification accuracy while obviating the need for copious training data. We train ViTAtr using exclusively non-stressed data without battlefield-simulating data augmentation with the aim of exploring classifier behavior and improving classifier performance in stressed imaging conditions. ViTAtr is shown to improve generalization in the face of image degradation caused by various environmental and anthropogenic stressors. Experimental results suggest that ViTAtr achieves an \(\approx \) 3.7%–4.7% improvement in classification performance over the baseline model (i.e., one that does not incorporate any attributes). Furthermore, in the most stressed environment (75% stress level), ViTAtr showed a significant (10%) improvement in overall classification performance compared to the baseline model.