In the process of English speaking assessment, it is necessary to use the evaluation system for analysis to improve the accuracy of the assessment. However, its architecture is complex, involving relatively much data, and complex pronunciation words and other content, so it needs to be improved with the help of learning intelligent analysis methods. This paper proposes an intelligent machine learning method to analyze English speaking assessment. The results show that the method can improve the effect of English assessment, and the recognition rate of pronunciation reaches more than 80%, shortening the evaluation time and optimizing the whole assessment. At the same time, the multi-architecture system such as language pronunciation score is analyzed to form the logic between the architectures, so that its rationality reaches more than 75%. Therefore, machine learning analysis methods can evaluate spoken English, optimize the original evaluation system, and realize the integration of framework and function.