<p>Maintaining a regular daily activity routine is essential for overall health and well-being. Wearable sensors offer a convenient way to track daily activities, but accurately identifying a wide range of activities remains a challenge for existing methods. These methods relied heavily on feature selection approaches in CNNs, which were not only tedious but also struggled to accurately identify activities with similar patterns and, also fails to find the long-range dependencies. The introduction of transformers in CNNs addresses these limitations, enabling more efficient and precise recognition of activities even when patterns are similar. In this paper, a novel Human activity recognition model that combines CNN with a Residual Transformer Network (RTN) is proposed. This model incorporates two key improvements: a sparse attention technique applied at multiple scales to capture complex spatial patterns, and an adaptive fusion. Sparse attention allows the model to concentrate on the most critical parts of the spectrogram at different scales and directions. This approach aids the model in distinguishing between activities with similar overall patterns by capturing subtle differences. Additionally, it enhances the model’s ability to recognize long-range dependencies. An adaptive fusion technique is introduced to combine features from CNN and RTN, with a dynamic scoring function optimally weighting each feature stream. This ensures balanced contributions from spatial patterns and temporal dependencies for accurate activity classification. Experiments conducted on the HARTH, HU-HAR, and HuGaDB datasets demonstrate impressive results with higher accuracies of 94.82%, 98.94%, and 96.33%, respectively. These findings highlight the effectiveness of the proposed method for accurate activity detection and continuous health monitoring applications.</p>