Cooperative behavior is commonly understood as that which is conducive to the good of the group: it is increasingly seen as a crucial component of advancing the capabilities as well as mitigating the harms of multi-agent AI systems [6, 10, 21]. Yet an “I’ll-know-it-when-I-see-it” approach is often taken when evaluating the cooperativeness of a sequence of actions, and even when cooperation is formalized, the definitions lead to category errors, conceptual confusions, and erroneous conclusions [11, 22, 52, 56]. We propose a formal measure of cooperation in stochastic games that avoids these pitfalls by being counterfactually contrastive, contextual, and customizable: in particular, cooperation is defined in contrast to the outcome that a self-interested actor would have effected in a similar circumstance, in the context of other agents’ behavior, and within a specified time and space horizon. This measure is simple to compute: it is dependent only on solving a reduction of the multi-agent game to a single-agent Markov decision process. We apply this measure to a diverse pool of behaviors in a number of mixed-motive social dilemmas and sequential predator-prey environments that have been studied in the multi-agent systems literature [4, 15, 26, 34, 36]. Our results demonstrate the importance of defining cooperation clearly, and provide a useful metric for builders of cooperative systems to use when establishing the cooperative nature of the system behavior.