Many criteria of algorithmic fairness have been put forth in the current literature, of which the two most important ones are arguably Equalized Odds and Calibration. However, despite their plausibility, Kleinberg et al. (2017) have shown that they are inconsistent outside of trivial cases, so at least one must be rejected in non-trivial cases. Recently, Eva (2022) propounded a weakening of Calibration, called Base Rate Tracking, though not necessarily in response to the preceding impossibility result. Regardless, even though it is weaker, Stewart et al. (2024) have shown that it is still inconsistent with Equalized Odds outside of trivial cases. In this paper, I propose a weakening of Base Rate Tracking I call Comparative Base Rate Tracking that is consistent with Equalized Odds even in non-trivial cases, yet this weakening is still quite substantial nonetheless. Importantly, I show that Comparative Base Rate Tracking can diagnose several deficient cases that cannot be diagnosed by other recently proposed criteria of fairness, like Spanning by Nielsen and Stewart (2024a) and Spacing by Eva (2024).
