<p>Careless responding is a pervasive problem that can produce misleading research findings; as a result, developing effective prevention strategies has important practical implications. The current research focuses on the effectiveness of the most widely used prevention strategy: participant warnings. Participant warnings are typically delivered via text placed at the beginning of the study questionnaire. This text warns participants that the questionnaire includes methods that detect careless responding. Warnings may threaten punishment for engaging in careless responding (severe warnings)—typically via loss of incentive—or they may omit such threats (benign warnings). While previous research indicates that warnings can reduce careless responding, these findings provide only a partial test of the effects of warnings, as careless responding is just one aspect of measurement quality. The current research extends prior studies by examining the effects of warnings on survey measures’ <i>construct validity</i>. Specifically, we conducted two randomized experiments (total <i>N</i> = 581 participant-informant dyads) in which we assigned each participant to one of three conditions: (a) control, (b) benign warning, or (c) severe warning. Participants completed self-report personality measures and recruited informants who completed measures of the participant’s personality and participant-directed criteria. This design allowed us to examine participant-reported measures’ convergent and criterion-related validity. Within both studies, this manipulation produced minimal effects, thus raising doubts about the effectiveness of warnings. Given our findings, practitioners and researchers should not rely solely on warnings; instead, they should explore other means of improving data quality such as incentives for attentive responding and participant prescreening.</p>