Computer Vision-Based Air Marshal Hand Signal Recognition for Air Mobility

Aircraft marshaling is a key visual procedure for the ground staff signals which directs the pilot to safely dock the aircrafts in the right position. With recent advancements in the technology, autonomy in aviation transportation has transformed into unmanned aerial vehicle (UAV) and Urban Air Mobility (UAM) transportation. In this regard, the marshal signals are understood by the pilots in the aircraft, but in the case of autonomous system, there is a need for the intelligence system to recognize and understand the signals given by the air marshal. This paper proposes a computer vision-based approach where in the marshal is detected and the pose of the marshal is estimated. A rule-based engine using the pose points deciphers and outputs the signal (pose) signaled by the air marshal. The proposed solution uses a trainable bag-of-freebies based method—YOLO v7 for human detection and human pose estimation. And air marshal signal recognition is built on top of this framework. Self-captured dataset has been used for testing purposes. The proposed method accurately recognizes marshal signals with an accuracy in range 85–94%. With the proposed method, it is possible to understand human signal for the docking of UAV/UAM and other aerial vehicles.