Deep learning-based object detectors are vulnerable to adversarial examples. For visible-infrared object detectors, existing cross-modal adversarial attack methods often fail to achieve high efficiency and transferability. To overcome these issues, we propose CAMPatch, a cross-modal adversarial patch attack method based on class activation map (CAM). Our CAMPatch is a two-stage patch optimization method, which can simultaneously evade visible-infrared object detectors. First, the regions of interest are generated by CAM to obtain an initial adversarial patch. Then, the differential evolution is introduced to iteratively optimize patch shapes under CAM guidance. The experiments demonstrate that CAMPatch not only improves the success rate of adversarial attacks but also has high efficiency and transferability.