The widespread availability of smartphones, combined with advancements in embedded sensing technology, has spurred a variety of applications in areas such as fitness, healthcare, environmental health and safety monitoring, and ambient assisted living. Recently, there has been a growing focus on recognizing daily human physical states, which is crucial for smart surveillance, home automation, and support for patients, the elderly, and individuals with special needs. This paper presents a novel approach, termed G-SwinHAR, and investigates its performance for hierarchical vision-based human activity recognition. Our method first transforms time-series signals from smartphone sensors into images using the Gramian angular field method, then applies a Swin transformer for hierarchical fusion of visual feature maps. We conducted a series of ablation and comparative studies on the UCI HAR and WISDM datasets. Besides memory reduction due to the shift-window multi-head self-attention mechanism, the results demonstrate that G-SwinHAR outperformed other benchmark methods that are based on convolutional neural networks.