Advancements in deep learning technology have significantly enhanced the accuracy and processing speed of pedestrian detection models based on deep neural networks. However, the vulnerabilities and lack of robustness of these networks make the models susceptible to malicious attacks, with backdoor attacks being a typical example. In a backdoor attack, an attacker implants a backdoor during the model training phase, causing the model to execute predefined malicious actions when triggered under specific conditions. In this paper, we present a novel backdoor attack method for fooling pedestrian detection models, named RetroreflectionBA. We use retroreflection as a trigger to attack these models. Our approach involves simulating and mathematically modeling the retroreflection pattern, then using this pattern to create poisoned samples and train a backdoored model. Under strong lighting conditions at night, the backdoor is strategically activated, causing the pedestrian detection model to make incorrect decisions. Due to the widespread presence of retroreflective materials in real-world scenarios and their subtlety during the day, this backdoor is highly stealthy and easy to deploy. Our study demonstrates that once our attack successfully implants the backdoor into the target model, it can deceive the model into failing to detect any pedestrians marked with our trigger pattern. Extensive evaluation on two datasets and four detectors validates the effectiveness and stealthiness of our backdoor attack.