Yellow Alert messages on Changeable Message Signs (CMS) aim to prevent harm and aid suspect apprehension after severe hit-and-run incidents by providing vehicle information. However, these alerts may distract drivers, altering their behavior and focus. This study evaluates the safety impact of Yellow Alerts using a driving simulation, analyzing speed variance, lane changes, headway distance, and participants’ recall and responses. Results show that Yellow Alerts significantly increased speed variance, lane changes, and headway variance. The presence of a suspected vehicle further intensified erratic behaviors, such as chasing or scanning for the vehicle. These findings emphasize the need to carefully balance public safety benefits with potential driver distraction risks when deploying Yellow Alerts.