Relationships Between Protocol Features and Engagement Rates in Studies of Youth Substance Use Using Mobile Technology

Ecological momentary assessment (EMA) generally involves short, repeated, real-time sampling of affect, cognitions, contexts, and behaviors, often with the aid of smartphone technology. This method allows for self-reports to be made while the respondent is in their everyday setting. The benefits include increased ecological validity and reduced response biases. These advantages are especially salient in youth substance use research where relevant events are episodic and variable in nature. One issue with current EMA literature is the lack of consistency in reporting study design features that may impact participant engagement. The aim of this study is to assess the extent to which a wide range of EMA study design characteristics, including protocol length, assessment frequency, incentive schedule, and length of time to complete assessments, are related to retention and response rates. A literature review related to mobile technology self-report assessment, substance use, and youth populations (ages 15–25) was conducted. A total of 90 papers were ascertained for analysis. Papers were coded for various design characteristics and engagement rates. A series of beta regressions were estimated to evaluate the association between design characteristics and response and retention rates. In unadjusted models, the number of reminder signals was associated with retention rate (b = −0.637, P = 0.002), and the number of assessments per day was associated with response rate (b = −0.065, P = 0.003). These were the strongest associations that were also found to be present after adjustment for covariates. Among adjusted models, lottery-style incentive systems had the strongest association with retention rate (b = 0.573, P = 0.003), and the number of incentive thresholds had the strongest relationship with response rate (b = −0.115, P = 0.027). Future confirmatory research employing experimental designs to explore the impact of study characteristics on participant retention and response rates in youth substance use studies should include the evaluation of the number of assessments per day, reminder signals, and incentive systems. This study highlights the low level of adherence to existing reporting guidelines and the need for greater transparency.