While convolutional neural networks (CNNs) are extremely popular and effective for image classification, we must understand the level of confidence in their predictions before using them to make decisions. We leverage probability theory and the generative capabilities of autoencoders to develop a probabilistic and reconstruction-based competency estimation (PaRCE) method. Through this approach, we obtain a human-interpretable score that captures multiple facets of predictive uncertainty arising in CNN models. We compare our method to existing approaches for uncertainty quantification and out-of-distribution (OOD) detection and find that our method can best distinguish between correctly classified, misclassified, and OOD samples with anomalous regions, as well as between samples with visual image modifications resulting in high, medium, and low prediction accuracy. We then describe how to extend our approach for anomaly localization tasks and demonstrate the ability of our approach to distinguish between regions in an image that are familiar to the perception model from those that are unfamiliar. We find that our method generates interpretable scores that most reliably capture a holistic notion of perception model confidence.