<p>The isolation system of high-end equipment such as aircraft engines usually adopts a composite design scheme that integrates specific nonlinear passive components and active control links. Its dynamic behavior is complex and prone to instability under internal resonance conditions. To investigate this issue, this paper establishes a fractional order Smooth and Discontinuous (SD) oscillator model controlled by Negative Derivative Feedback (NDF), which takes into account the memory dependent characteristics of metal rubber. This study first derived the first-order approximate analytical solution of the system using the multiscale method, and analyzed the stability of the slowly varying manifold using the Routh-Hurwitz (RH) criterion, thus revealing the parameter excitation mechanism of internal resonance. Based on this analytical framework, numerical analysis elucidates the influence of system parameters on amplitude frequency response and basin of attraction morphology. Finally, with the goal of suppressing internal resonance and expanding the stability domain, we optimized the NDF control parameters using simulated annealing algorithm. The results indicate that the unstable vibration of the optimized system has been effectively suppressed, and the stability domain has not been reduced. This study provides a theoretical solution for the dynamic design and control optimization of strongly nonlinear isolation systems under internal resonance conditions, which is of great value in improving the reliability of high-end equipment.</p>