Multimodal IR and RF Based Sensor System for Real-Time Human Target Detection, Identification, and Geolocation

The Dynamic Data Driven Applications System (DDDAS) paradigm incorporates forward estimation with inverse modeling, augmented with contextual information. For cooperative infrared (IR) and radio-frequency (RF) based automatic target detection and recognition (ATR) systems, advantages of multimodal sensing and machine learning (ML) enhance real-time object detection and geolocation from an unmanned aerial vehicle (UAV). Using an RF subsystem, including the linear frequency modulated continuous wave (LFMCW) ranging radar and the smart antenna, line-of-sight (LOS) and non-line-of-sight (NLOS) friendly objects are detected and located. The IR subsystem detects and locates all human objects in a LOS scenario providing safety alerts to humans entering hazardous locations. By applying a ML-based object detection algorithm, i.e., the YOLO detector, which was specifically trained with IR images, the subsystem could detect humans that are 100 m away. Additionally, the DDDAS-inspired multimodal IR and RF (MIRRF) system discriminates LOS friendly and non-friendly objects. The whole MIRRF sensor system meets the size, weight, power, and cost (SWaP-C) requirement of being installed on the UAVs. Results of ground testing integrated with an all-terrain robot, the MIRRF sensor system demonstrated the capability of fast detection of humans, discrimination of friendly and non-friendly objects, and continuously tracked and geo-located the objects of interest.