The Factor graph algorithm has an edge in address the issue of aerospace multiple navigation system scheme configuration due to its plug and play capability. The large dynamic flight environment for aerospace vehicles will bring great challenges to navigation system, causing noise characteristics to be inconsistent with the model, thus making fusion algorithm estimation biased. Aiming at the problem of navigation accuracy reduction, a multi-source fusion navigation method for aerospace vehicles based on adaptive factor graph optimization (AFGO) was proposed in this paper. Adaptive particle swarm optimization (APSO) was introduced for adaptive estimation of the measurement noise covariance matrix. The proposed method performed well in simulated environment. The AFGO method can adaptively optimize the covariance matrix of measurement noise. Therefore, when the measurement noise changes greatly due to unknown environmental factors during the flight of aerospace vehicles, the proposed method can estimate the actual noise, thus improving the accuracy and availability of the navigation system ultimately.