This paper presents a post-hoc method for detecting out-of-distribution (OOD) examples in deep neural networks by utilizing multi-class output calibration misalignment. OOD detection is an essential subfield of outlier detection, which aims to classify new instances as either conforming to the training set distribution or being anomalous. Our method quantifies the difference in output probability calibration for individual classes by utilizing a one-vs-rest approach to the normalized total probability mass over all class calibrations. Furthermore, we leverage the magnitude of the probability mass misalignment between that of the combined probabilities of class pairs over the expected sum of their individual calibrated probabilities. Our results show that this discrepancy and probability misalignment can effectively be used to identify OOD examples for neural networks, as calibration alignment trained on in-distribution examples behaves anomalously when presented with outliers. Additionally, our prediction-based OOD approach can outperform common embedding-based anomaly detection methods on standard computer vision datasets.