To address the challenge of autonomous navigation under the constraints of signal interference and small payloads, this paper presents a novel approach involving the observation of Resident Space Objects (RSOs) using optical cameras for autonomous orbit and attitude determination. With the increasing number of RSOs, the orbital data of RSOs is becoming increasingly rich, which can serve satellite navigation tasks. Initially, we establish an autonomous navigation model, comprising orbital attitude state models and optical observation models. Subsequently, an analysis of the rank and condition number of the Gram matrix is conducted to evaluate the observability of this navigation system. Finally, leveraging optical observation data of RSOs and filtering techniques, orbit and attitude estimation for small satellites are realized, which further demonstrates that using optical observations of RSOs is effective and feasible for estimating satellite orbit and attitude.