Autonomous vehicles are revolutionizing transportation systems by improving safety, efficiency, and reliability. However, their ability to interact seamlessly with human operators, such as traffic police officers, remains a critical challenge, especially in dynamic and unpredictable urban environments. This work presents a robust framework for gesture recognition of traffic police personnel using pose estimation, focusing on their role in directing vehicles. Pose estimation identifies key joint positions such as the shoulders, elbows, and wrists, creating a skeletal representation of the traffic personnel. YOLOv7 (You Only Look Once) pretrained model is used for pose estimation for extracting key point information from visual inputs. Convolutional Neural Networks (CNN), LSTM (Long Short-Term memory), LSTM-CNN with attention mechanisms, GRU (Gated Recurrent Unit), GRU-CNN and GRU model with attention are used for gesture prediction based on the extracted pose data. LSTM and GRU models are very good at finding long-term dependencies in sequential data, making them ideal for using in dynamic pose changes across frames. CNN layers efficiently extract spatial features, while LSTM/GRU layers excel at learning temporal patterns. Combining both enhances model performance. The optimal model for gesture prediction is decided based on the classification accuracy and computational efficiency. The proposed system aims to enhance autonomous vehicle perception by accurately interpreting traffic personnel’s hand gestures, thereby improving decision-making capabilities and ensuring operational safety. To evaluate the models, Chinese traffic police gesture dataset in video format which has 8 different hand gesture commands is used. The GRU-CNN model with attention mechanism has a better accuracy compared to the LSTM, LSTM-CNN models. It is inferred that GRU-CNN based attention model performs better than other similar models and can be efficiently used for recognizing the gesture shown by the traffic personnel. It is also inferred that YOLO v7 model can be used for creating pose estimation inputs for these models very efficiently. In this paper GRU-CNN and GRU-CNN with attention mechanism based models produce an accuracy of 95.02% and 95.23% respectively.