<p>Unequal helix angle end mills effectively suppress machining chatter, demonstrating distinct advantages in precision machining applications. Fluctuations in instantaneous cutting forces during milling constitute a primary cause of chatter in machining systems. To enhance prediction accuracy, accurate prediction of instantaneous forces is essential. This paper establishes an instantaneous milling force prediction model for unequal helix angle end mills considering tool runout and tool actual trajectory. Subsequently, to address the issue that traditional coefficient calibration methods are not applicable to unequal helix angle end mills, a coefficient calibration method based on oblique cutting transformation theory and particle swarm optimization algorithm is proposed. Finally, the prediction model was validated through side milling on Inconel 718 nickel-based superalloy. The results indicate that the maximum error between predicted and measured milling forces is 6.54%, which demonstrates this model’s high accuracy.</p>