A new criterion for assessing fairness of AI models in medical imaging is proposed. The key idea is to control for disease severity, which as a mediator, affects the presentation of disease in medical images, and hence the performance of AI algorithms. Existing fairness criteria such as equalized odds do not capture this effect, as is illustrated by an example. Additionally, a new metric is proposed based on the information theoretic notion of adjusted mutual information. The metric is easy to compute, captures the overall bias of an AI model, and can be used to compare multiple models, which is illustrated using an example. In this example, three chest X-ray classification models, trained on NIH, CheXpert and PadChest datasets, respectively, are used to predict on a subset of MIMIC-CXR cases, for which the severity scores of pulmonary edema are available, and the bias of the three models are computed and compared along two sensitive attributes: sex and ethnicity.