<p>Financial institutions increasingly rely on predictive models for automated mortgage underwriting. Understanding the strengths and weaknesses of these models is thus of paramount importance for the stability of the modern financial system. Little is known about the predictive accuracy of these models; perhaps more importantly, the mechanisms underlying the failure of predictive models are not well understood. In the first portion of this paper, we utilize millions of loan-level servicing records for mortgages originated between 2004 and 2016 to construct models of mortgage performance that mimic the models currently used by financial institutions to support residential mortgage originations. We find that these models can be wildly inaccurate when used to predict loan performance in out-of-time samples, a phenomenon that we refer to as model failure. Importantly, we provide evidence that predictive model failure was not unique to loans originated during the early 2000s subprime boom; rather, model failure is the rule, not the exception. We use the Panel Study of Income Dynamics in the second part of our paper to provide evidence that one potential mechanism driving model failure is the intertemporal heterogeneity in the relationship between variables that are frequently used to predict mortgage performance and the realized post-origination path of variables that have been shown to trigger mortgage default. Our findings imply that model instability is a significant source of risk for lenders, such as financial technology firms, that rely heavily on predictive statistical models and machine learning algorithms for underwriting and account management.</p>