<p>Magnesium alloys provide a variety of advantageous characteristics, including a high strength-to-weight ratio, superior corrosion resistance, and exceptional machinability. The machinability of magnesium alloys makes them a feasible solution in biomedical applications. The current study minimizes the tool vibration and surface roughness of the magnesium alloy matrix composite during turning operations by selecting optimum machining parameters. A stir casting procedure was used to create the biocompatible composite alloy, which consisted of 2% nano-alumina (Al₂O₃), 6% micro-tin particles, and 92% AZ31 magnesium alloy. The turning experiments are conducted on a CNC lathe, considering the depth of cut, feed rate, and spindle speed as parameters. The surface roughness and tool vibration are measured for 20 experiments that are planned using the Central Composite-based Response Surface Methodology (RSM). Regression models are developed to predict the surface roughness and tool vibration, and then the adequacy of the model is tested using ANOVA. The results revealed that the tool vibration in axial, radial, and tangential directions has the same magnitude while machining magnesium alloy. The results also indicate that when turning magnesium alloy, spindle speed predominantly affects surface roughness, while depth of cut mostly influences tool vibration negatively. A multi-objective optimization process is implemented using RSM and Pareto optimality to minimize tool vibration and surface roughness by selecting optimum tuning parameters. The predictive ANN model is found to be effective for determining tool vibration and surface roughness, as indicated by the gradual decline of the MSE plot, which flattens near 0.00018295.</p> Graphical abstract <p></p>