<p>The compressive flow stress of a novel Mg-1.8Nd-0.4Zr-0.3Ca alloy was measured using a hot compression testing machine at temperatures between 330 ℃ and 460&#xa0;°C and strain rates ranging from 0.001 to 1&#xa0;s⁻¹. The flow stress data were utilized to develop both conventional and strain-compensated constitutive models in conjunction with evaluating the deformation activation energy. Processing maps were further established at various strain levels. The results demonstrate that the established constitutive model effectively predicts flow stress behavior. The deformation activation energy decreases with increasing strain, reaching 170&#xa0;kJ/mol under peak stress conditions. Results from the dynamic material model reveal that the stable processing window for this alloy ranges from 380 to 460&#xa0;°C and at strain rates of 0.005 to 0.05&#xa0;s⁻¹. Microscopic characterizations and analyses revealed the presence of non-basal slip, continuous dynamic recrystallization, &lt; 0001&gt;//CD texture, and dynamic precipitation. Additionally, unstable regions (380&#xa0;°C, 1&#xa0;s⁻¹) in the processing map exhibited high-density dislocations, twins, deformation bands, and strong textures, indicating microstructural heterogeneity that may induce local stress concentration and flow instability. The instability modes of Mg-Nd-Zr-Ca alloy under extreme processing parameters provide theoretical guidance for deformation process design and manufacturing parameter determination.</p> Graphical Abstract <p></p>