Scientists and inventors set the direction of their work amid evolving questions, opportunities and challenges, yet the understanding of pivots between research areas and their outcomes remains limited1–5. Theories of creative search highlight the potential benefits of exploration but also emphasize difficulties in moving beyond one’s expertise6–14. Here we introduce a measurement framework to quantify how far researchers move from their existing work, and apply it to millions of papers and patents. We find a pervasive ‘pivot penalty’, in which the impact of new research steeply declines the further a researcher moves from their previous work. The pivot penalty applies nearly universally across science and patenting, and has been growing in magnitude over the past five decades. Larger pivots further exhibit weak engagement with established mixtures of prior knowledge, lower publication success rates and less market impact. Unexpected shocks to the research landscape, which may push researchers away from existing areas or pull them into new ones, further demonstrate substantial pivot penalties, including in the context of the COVID-19 pandemic. The pivot penalty generalizes across fields, career stage, productivity, collaboration and funding contexts, highlighting both the breadth and depth of the adaptive challenge. Overall, the findings point to large and increasing challenges in effectively adapting to new opportunities and threats, with implications for individual researchers, research organizations, science policy and the capacity of science and society as a whole to confront emergent demands.
