DeepSense: test prioritization for neural network based on multiple mutation and manifold spatial distribution

Deep learning systems have been used extensively in several fields in recent years, but deep neural network (DNN) can also make incorrect decisions and lead to significant losses. Testing tasks based on DNN systems often require annotation of test data to obtain oracle information. However, collecting and annotating large amounts of disparate data from application scenarios is very expensive and time consuming. Thus, we propose DeepSense, an effective neural network test prioritization technique, to select more test inputs that reveal neural network faults as early as possible in the unlabeled test dataset. DeepSense considers the full range of model fault detection capabilities, including faults near the boundary and near the centroid of class. To be specific, we first designed multiple mutation features and extracted them from the neural network model and input samples based on different mutation operators. Then, the sample embedding features are extracted to construct an undirected weighted graph, and a random walk is performed to calculate the distance similarity and manifold similarity, and then spatial nearest neighbor features are designed and extracted. Finally, MLP is used to combine multiple mutation features and spatial nearest neighbor features to predict the input sample fault revealing ability and set priorities accordingly. We evaluate DeepSense on four popular image datasets, and results show that DeepSense significantly outperforms existing test input prioritization techniques.