The rapid growth of low-altitude economy has made low-altitude airspace safety a critical issue. The existing ground-base detection systems encounter several challenges, including high deployment costs, limitations imposed by commercial frequency bands, and low recognition rates in complex environments. To enhance anomalous UAV recognition precision, we propose SkyPatrol, a device-edge collaborative framework for aerial peer perspective vision based anomalous UAV recognition. Specifically, we take trusted mission oriented or specialized UAVs as patrol UAVs to capture imagery of UAVs within their field of view. Then, utilizing periodically reported peer-view imagery, the ground-based intelligent detection system predicts UAVs’ physical positions, and recognizes anomalous UAVs by correlating positional data periodically reported by cooperative UAVs. For precise position prediction in SkyPatrol, we predict the visual features of observed UAVs (e.g., longitude, latitude, altitude) in the computed overlap area of images from multiple patrol UAVs based on YOLO model; then, we design a TriLocNet to predict their position. Extensive experiments show that in SkyPatrol, the mean squared error of the predicted position is 0.186. In comparison experiments, SkyPatrol outperforms classical methods across different lighting conditions (daytime, dusk, and night) and various distribution densities of UAVs.