With the continuous development of Internet of Things (IoT) and Artificial Intelligence (AI) technologies, edge computing is emerging as a near-data source computing model. Due to the limited computing resources of edge devices, it is essential to measure and analyze the performance of neural network models to ensure efficient use of these resources during inference on edge devices. However, most existing prediction techniques are based on the model or kernel operator level. They require access to the specific code of the neural network model, making them unsuitable for black-box models. Additionally, most current prediction technologies necessitate large-scale data sample collection, which incurs substantial time overhead on edge devices. In response to these challenges, this paper proposes a prediction algorithm based on machine learning that does not require internal details of the neural network model. The algorithm leverages Bayesian Optimization to optimize the hyperparameters of the training model and employs a Gaussian process regression model to guide targeted data sampling, thereby reducing overall training costs. Furthermore, experimental results verify the algorithm’s stability and excellence in model performance prediction, as well as its feasibility in constructing an accurate performance model with a small amount of sample data.