Vehicular networks based on distributed machine learning have gained extensive research attention due to their outstanding performance compared to traditional methods. Specifically, machine learning systems can output vehicle control signals based on inputs from the object detection models. However, it has been demonstrated that machine learning is susceptible to security issues, where adversarial samples may mislead models into producing incorrect outputs. In this paper, we propose a novel adversarial patch attack against transformer-based object detection models. Our attack can make the object of a specific class invisible to object detection models. To enhance the attack performance and transferability of the adversarial patches, we designed the Gradient Self-Ensemble module and Transformer Masking Matrix module. These modules obscure parts of the inputs to each encoder and decoder and aggregate the outputs of all decoders within the model to generate adversarial samples, significantly improving the success rates of black-box attacks. We successfully suppressed the model’s inference capabilities and deceived multiple transformer-based object detection models. Our work highlights the vulnerability of object detection models in vehicular networks to adversarial patch attacks in both digital and physical domains.