In order to improve the effect of oral English teaching and improve the intelligent evaluation effect of oral test, this paper develops a voiceprint recognition network based on time-frequency domain feature fusion, and fuses the time-delay neural network branch and the traditional acoustic feature processing branch feature map and then sends it to the voiceprint embedding extraction network, which effectively improves the model’s ability to capture time series and frequency information. Meanwhile, the kernel attention unit is used in the network instead of ordinary convolution, which makes the network capture feature information of different scales more accurately and makes it more flexible and efficient when dealing with complex time-frequency domain features. Finally, combined with experiments, this paper verifies that the model has a good effect in oral English correction and examination evaluation, which provides great convenience for teaching organizers and managers and can further improve the organization and management efficiency of oral English teaching.