Since the performance of image classification algorithm based on deep learning is closely related to its network complexity and the training data, we propose a new experimental method to measure the effectiveness of training data sample size. An image classification algorithm is regarded as a decision making units with network parameters input and classification accuracy output according to a certain dataset. This new method, which measures the algorithm’s efficiency, is quite different with those traditional performance modeling approaches. With the help of data envelope analysis theory and Malmquist index, this new method can deal various evaluation metrics naturally. In order to avoid complex performance modeling, the variable scale to return model is chosen to calculate the adjacent-period distance which reflects the effect of training data. Then the distance function is decomposed into the technical change and the efficiency change. The former measures the effect of training data on those Pareto optimal algorithms, while the later gives each algorithm’s efficiency of data using. At last, a traffic light image classification example shows that the proposed method is flexible and convenient.