DeepTD: Diversity-Guided Deep Neural Network Test Generation

Coverage-guided Fuzz Testing (CGF) techniques have been applied to deep neural network (DNN) testing in recent years, generating a significant number of test samples to uncover inherent defects in DNN models. However, the effectiveness of CGF techniques that utilize structured coverage metrics as coverage criteria is currently being questioned. A few unstructured coverage metrics, such as surprise adequacy, only take into account the diversity of the test samples against the training set, while ignoring the diversity of the test samples themselves. In addition to this, the existing surprise adequacy metrics have some limitations in their applications. Therefore, this paper proposes DeepTD, a diversity-guided deep neural networks test generation method. Firstly, DeepTD selects high-loss test samples from each class on average, ensuring these test seeds possess a strong ability to reveal model errors. Then, DeepTD transforms these test seeds to enhance the diversity of the generated samples. Finally, Cluster-based Surprise Adequacy is designed to guide the generation of test samples. To evaluate the effectiveness of DeepTD, six DNN models are selected as subjects, covering two well-known image datasets. Experimental results demonstrate that the Cluster-based Surprise Adequacy outperforms the two existing metrics not only in computational efficiency but also in discovering more model defects. What’s more, the test samples generated by DeepTD are on average 6.04% and 3.24% more effective for model retraining in MNIST and CIFAR10 compared to baseline methods, respectively.