Purpose <p>Targeting advanced engineering applications of the Earth observation, this study resolves critical challenges in dynamics modeling of large-aperture flexible appendages and precision formation control.</p> Methods <p>A thin-film diffraction imaging system is designed to reduce weight, enhance resolution, and improve revisit capability. A rigid-flexible spacecraft dynamics model is derived using dual quaternion, and a fractional-order operator is employed to design an attitude-orbit integrated sliding mode controller. Asymptotic stability is mathematically proven, with a fast terminal sliding mode controller (FTSMC) introduced for comparative analysis of the fractional operator's impact on control performance.</p> Results <p>Theoretical analysis confirms the system's asymptotic stability, while numerical simulations validate the dynamics model and controller effectiveness.</p> Conclusion <p>The utilization of dual quaternions provides an effective framework for characterizing the cross-coupling phenomena of attitude, orbit, and vibration in rigid-flexible coupled spacecraft. Fractional-order operators introduce additional tunable parameters into traditional control frameworks, thereby enhancing control flexibility and robustness.</p>