The Application of Randomized and Quasi-Experimental Designs in Federal Government Program Evaluation

The chapter aims to share the knowledge of methodological approaches used to assess the effectiveness of labour market programs in the federal government setting both randomized controlled trials (RCTs) and quasi-experimental designs. A few cases of experimental studies are discussed to illustrate some of the key challenges of implementing RCTs in the Government context, including, low external validity, ethical considerations and the timing of the RCT roll-out. Given these challenges, the chapter highlights that quasi-experimental methods could help produce robust evaluation findings and recommendations to support decision making at the federal level. For example, the chapter outlines the use of matching techniques combined with the difference-in-differences method to control for observable and time-invariant unobservable factors in estimating the causal effects of labour market programs under the Employment and Social Development Canada (ESDC) portfolio. Other quasi-experimental designs are also presented, such as, the regression discontinuity design used to provide unbiased estimates of the causal effects for the Employment Insurance program, precisely the impact calculation of extending benefits at the margin. The chapter then demonstrates the application of causal machine learning to provide granular evidence about the program impacts on employment outcomes by gender, age group, disability status, indigenous, visible minority, and immigration status. Given the importance of complementary approaches to incremental impact analysis, the chapter also provides an overview of qualitative methods used in the ESDC context. Finally, examples are shared on how impact evaluations contributed to evidence-based decision making and recommendations are given for future policy and process considerations.