Since image recognition technology was confirmed to be vulnerable to attack, research on adversarial attack methods has emerged one after another. There are also many studies on the adversarial patch attack methods of the mainstream object detector YOLO (You Only Look Once) series. However, with the emergence of more advanced object detectors such as YOLOv5 [1], these existing attack methods have lost their effectiveness in both digital and physical attacks. To solve this problem, in this work, we propose a new adversarial attack method, InviSpell, for the new network structure, which designs a new loss function to generate adversarial patches based on the network structure of the YOLOv5 model. In this method, a new optimization strategy is proposed that uses the target confidence to adjust the optimization weights. The experiments show that the adversarial patch generated by our method can reduce the mean average precision of YOLOv5 from 71.24% to 2.86%. Our method has good attack effects in both the digital and physical worlds on YOLO v2 to v5 and Faster R-CNN. Moreover, the posters and T-shirts printed with the adversarial patch have good attack effects on the object detector and good transferability between different detectors and training datasets.