Neuron Coverage Maximization for Effective Test Set Construction with Respect to the Model

Real world data is not stationary and thus models must be monitored in production. One way to be sure in a model’s performance is regular testing. If the labels are not available, the task of minimizing the labeling cost can be formulated. In this work, we investigate and develop various ways to construct a minimum test set for a given trained model, in a fashion where the accuracy of the model calculated on the chosen subset is as close to the real one as possible. We focus on the white box scenario and propose a novel approach that uses neuron coverage as a observable functional to maximize in order to minimize the number of samples. We evaluate the proposed approach and compare it to Bayesian methods and stratification algorithms that are the main approaches to solve this task in literature. The developed method shows approximately the same level of performance but has a number of advantages over its competitors. It is deterministic, thus eliminating the dispersion of the results. Also, this method can give one a hint on the optimal budget.