An enhanced vision transformer with scale-aware and spatial-aware attention for thighbone fracture detection

Vision transformers (ViTs) have recently outperformed convolutional neural networks (CNNs) across a variety of deep learning tasks. In the field of orthopedic medicine, the thighbone serves as a critical support structure for the lower body, and a timely and accurate diagnosis of its fractures is important to preventing lifelong walking disabilities. Despite the successes of CNNs in the computer-aided diagnosis of thighbone fractures, the potential of ViTs in this realm remains unexplored. Consequently, we initially explored the direct application of off-the-shelf ViT models on thighbone fracture detection but found the results did not fully satisfy the requirement of radiologists. To address this gap, we propose a one-stage hybrid method that combines enhanced vision transformers with the CNN attention mechanisms, specifically for thighbone fracture detection. Our method improves a pyramid vision transformer architecture and employs overlapping patch embedding to preserve the local continuity in X-rays. For dynamic feature fusion across spatial and scale dimensions, we use a series of attention mechanisms consisting of two distinct types: scale-aware attention and spatial-aware attention. These mechanisms can integrate feature maps output from the neck structure, thereby improving the representation of thighbone fractures. We validate the proposed method using a meticulously curated dataset of 4000 thighbone X-rays, annotated by experienced radiologists. Ablation studies confirm the effectiveness of each modification in our proposed framework. Experimental results show that our method achieves an average precision (AP) of 53.7% and an \(AP_{50}\) $A{P}_{50}$ of 87.0%, thereby surpassing all previous state-of-the-art methods in thighbone fracture detection.