Object detection techniques for automonous Unmanned Aerial Vehicles (UAV) relies on deep neural networks. However, research has shown that deep neural networks are vulnerable to adversarial attacks, especially physical adversarial attacks, which can lead to misdetections by the object detection. To address the issue of poor attack effectiveness of current physical adversarial attack methods in object detection with UAV scenarios, we propose a new scale-adaptive physical attack method. We design a scale-adaptive scheme that scales the patches according to masks constructed from ground truth values to accommodate multi-scale targets. Robust adversarial patches are constructed based on changes in the UAV perspective, distance, and brightness. A new loss algorithm is designed, optimizing the adversarial patches by considering more available information from the detected objects. Our experiments on the VisDrone dataset verify that in a white-box setting, we achieved an attack success rate of up to 81%. Furthermore, for transferred patches against DNN models with different initializations and architectures in a gray-box setting, we achieved attack success rates of up to 79.7% and 78.8%, respectively.