Cooperative Navigation is a crucial task in drone swarm combat. This paper conducts further investigations based on the research of a model-based reinforcement learning method, TEAMSTER, which is used in ad hoc teamwork for cooperative navigation tasks in drone swarm urban combat. Three different representative kinds of mapped urban combat domains, the center base district (CBD) domain, the sparse architecture district (SAD) domain, and the multi-facility district (MFD) are set as our experiment scenarios to explore various aspects of this method. The experiments conducted assess the stability, robustness and generalization capability of TEAMSTER in response to changes in environment complexity and team scale of the drone swarm cooperative navigation ad hoc teamwork. Results show that both environment complexity and team scale of the ad hoc swarm drone agents significantly impact the performance of TEAMSTER. Furthermore, the study demonstrates that TEAMSTER performances well across diverse team scales and various task domains compared with other baselines. Overall, this study is a meaningful exploration to broaden the scope of research on model-based reinforcement learning method which is conducive to enhanced the drone swarm ad hoc teamwork for cooperative navigation.