Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis

Recent research reveals many pitfalls when selecting a comparison group for difference-in-differences analyses of observational data. Recommendations from this research, although important, can be difficult to apply to a complex, real-world evaluation. In this study we explore the potential value of tailored simulation studies to inform comparison group designs for difference-in-differences analysis. We developed a simulation study mirroring the evaluation of Primary Care First (PCF), a primary care transformation model offered in select regions of the United States. We simulated primary care practices that volunteer for the intervention (intervention group) and two potential comparison groups: non-participating practices in PCF regions (within-market) and practices outside PCF regions (out-of-market). We assumed within-market comparison (i.e., non-participating) practices differ systematically from participating practices, whereas out-of-market practices are more similar to participating practices but experience different market-level outcome trends and shocks. We used Medicare fee-for-service spending data to quantify the likely magnitude of these forces. Using difference-in-differences analysis, we then estimated a hypothetical intervention’s impact relative to each comparison group. We compared the mean squared error (MSE) of these impact estimates between the two comparison groups. The simulation revealed a bias-variance trade-off: the greater variation in impact estimates for the out-of-market comparison group generally led to higher MSE than the greater systematic bias of impact estimates for the within-market comparison group. More generally, a bespoke simulation study grounded in real data can quantify forces relevant to comparison group design and offer guidance to researchers seeking the best approach for their own evaluations.