Conformal prediction for out-of-distribution time-series classification
摘要
We examine the problem of handling uncertainty in the predictions for out-of-distribution data (OOD) when classifying time series with distortions, such as gaps, that can occur due to operating conditions in production environments. This problem can find several application domains, including predictive maintenance. We focus on Conformal Prediction (CP) as a framework for handling the uncertainty in predictions for OOD time series that occurs due to distortions. Our study focuses on the potential impact of OOD time series on the performance of CP, by assessing the size and coverage of the resulting prediction sets. The motivation for this study is that neural networks, which are widely used for time-series classification, may suffer from overconfidence. This fact negatively impacts CP when faced with OOD data, because incorrect predictions with high confidence can have catastrophic consequences in high-risk applications. To alleviate this problem, we propose two model-agnostic methods: the use of various forms of label smoothing as well as the use of hybrid classifiers. Our experimental findings in the context of prominent time-series classifiers demonstrate that the coverage of CP may be maintained around a desirable level without needlessly expanding the size of the prediction sets.