The use of deep learning for classification tasks has shown great potential in medical applications. In critical domains as such, it is of high interest to have trustworthy algorithms which are able to tell when a reliable assessment cannot be guaranteed. Hence, detecting out-of-distribution (OOD) samples is a crucial step towards building a safe classifier. Following a previous study, showing that it is possible to classify breast cancer in point-of-care ultrasound (POCUS) images, this study investigates out-of-distribution (OOD) detection. Three different OOD detection methods were implemented and evaluated in this study: softmax score, multi-level energy score and deep ensembles. As in-distribution training data both standard ultrasound images and POCUS images were used and a separate POCUS data set was used for testing. All OOD detection methods were evaluated on three different OOD data sets, which are a mixture of synthetic data and real ultrasound data that represent different use cases for which OOD detection in automatic breast cancer classification is needed, covering a range of simple OOD cases, ultrasound images of poor quality and ultrasound images of non-breast tissue. The results show that the softmax score is inferior compared to the other methods at detecting OOD samples. The multi-level energy score performs superior on two of the OOD data sets. The deep ensembles perform superior on the OOD data set containing ultrasound images of poor quality with a 95% confidence interval for the area under the receiver operating characteristic curve of 97.2%–98.5%.