Physical activity, such as playing sports, is useful in maintaining a heathy mind-body complex. In this work, multiple wearable surface electromyography (sEMG) and accelerometer sensors have been employed to capture data for hand gestures of cricket umpire. The proposed approach will not only accurately interpret the umpire’s decision but may also be utilized during their training. Deep learning models consisting of convolutional layers and recurrent layers are finely tuned to process these different modality data for hand gesture recognition. Moreover, attention mechanism is utilized to enhance the efficacies of the models. The attention scores assist the network in focusing on selective regions of the input feature map, hence improving the performance of the network. The proposed convolutional recurrent neural network (CRNN) along with attention layer achieves an average accuracy of 97.1% for accelerometer data. This is around 2% higher than the accuracy attained by the same model without the attention mechanism. Similar observations are obtained for the sEMG data demonstrating the utility of attention mechanism in time-series data-based classification task.