As a key component of milling machine, the stiffness and quality of swing milling head directly affect the machining accuracy and stability of the milling machine. However, high stiffness and light weight are one of the key technologies to improve the performance level of swing angle milling head. Considering the limitation of response surface optimization, the beluga optimization algorithm is introduced and a multi-objective genetic algorithm optimization method based on improved BP neural network is proposed. Taking a certain model B-axis swing angle milling head as the test object, using DOE design sampling to establish the data set, and taking high stiffness and light weight as the optimization design indexes, we carry out the research on the optimization design of the swing angle milling head, and analyze the test results in finite element simulation test. As a result, the optimized milling head shell reduces the mass by 12.36% under the design requirement of static stiffness, while the dynamic performance is significantly improved, which verifies the reasonableness of the optimization method, and provides a way of thinking for the structural optimization design of the key components of the machine tool.