When working in a high-risk setting, having well calibrated probabilistic predictive models is a crucial requirement. However, estimators for calibration error are not always able to correctly distinguish which model is better calibrated. We propose the conditional kernel calibration error (CKCE) which is based on the Hilbert-Schmidt norm of the difference between conditional mean operators. By considering calibration error as the distance between conditional distributions, which we represent by their embeddings in reproducing kernel Hilbert space, the CKCE is less sensitive to the marginal distribution of predictive model outputs. This makes it more effective for relative comparisons than previously proposed calibration metrics. Our experiments, using both synthetic and real data, show that CKCE provides a more reliable measure of a model’s calibration error and is more robust against distribution shift.