Prediction Accuracy & Reliability: Classification and Object Localization Under Distribution Shift

Natural distribution shiftDistribution shift causes a performance deterioration for the visual perception computed by convolutional neural networks (CNNs). This can lead to safety–critical situations, when deploying CNNs to real-world traffic data. To facilitate a quantitative analysis of this deterioration, this work addresses: (1) investigating the effect of natural distribution shiftDistribution shift and weather augmentations on the performance of detection and confidence estimation, (2) evaluating model accuracy and confidence prediction for both object classificationClassification and its localization, and (3) benchmarking two common uncertaintyUncertainty quantification methods - EnsemblesEnsemble and compute-efficient variants of Monte-Carlo (MC) DropoutDropout. For this purpose, a novel dataset has been curated from publicly available autonomous driving datasets. The in-distribution (ID) data is based on cutouts of individual objects, for which both class and bounding box annotations are available. The six distribution-shift datasets cover adverse weather scenarios, simulated rain and fog, corner cases, and out-of-distribution data. A granular analysis of CNNs under distribution shiftDistribution shift allows to quantize the impact of different types of shifts on both, task performance and confidence estimation. Our novel key findings are: ConvNeXt-Tiny is more robust to distribution shiftDistribution shift than EfficientNet-B0; heavy rain degrades classificationClassification accuracy stronger than localization, whereas heavy fog has the opposite effect; integrating MC-Dropout into selected layers only has the potential to enhance task performance and confidence estimation, with the identification of these layers depending on the type of distribution shiftDistribution shift and the considered task.