Application of Image Captioning and Zero‑Shot Detectors for Building a Unified Map from Group Drone Imagery
摘要
This paper addresses the problem of merging images acquired by multiple unmanned vehicles to build a unified map of orchards in the Kursk region, Russia. Traditional mosaicking approaches rely on either geodetic sensors or dense matching of local visual features and perform poorly in orchards due to the repetitive structure of tree rows and frequent occlusions. We propose a method that relies on a semantic description of scenes: each drone uses an Image Captioning model to generate textual descriptions of key objects and then transmits these descriptions to other drones. Based on shared textual descriptions, open‑vocabulary (Zero‑Shot) detectors localise the corresponding objects, and their coordinates are used to estimate geometric transformations between images and construct the global map. The method was tested both in a simulator and on real data; experiments show a lower average map merging error compared with classical methods.