The dynamic nature of Autonomous Underwater Vehicle (AUV) ad hoc networks is influenced by the complexities of the ocean environment, unpredictable targets, and limited communication capabilities. In addition, limited communication hinders timely access to global information. Although researchers have acknowledged the impact of constrained communication on topology optimization, a detailed mathematical model to analyze these constraints specifically is lacking. This paper focuses on optimizing network topology in AUV ad hoc networks for cooperative search missions under communication constraints. Firstly, to analyze the effect of underwater acoustic communication quality on topology optimization more accurately, some mathematical models of underwater acoustic communication is established. Then we propose a distributed alternating optimization algorithm to facilitate efficient communication despite incomplete global information. Specifically, we introduce a hierarchical flooding self-organizing routing (HF-SOR) method for network routing optimization. This method relies solely on neighbor information, eliminating the need for global information, thus enhancing adaptability and robustness. The hierarchical routing method achieves network routing planning and stratifies the network based on the data transmission path. AUVs near the source AUV have a significant impact on the network, with closer proximity resulting in greater influence. Therefore, we design topology optimization rules based on hierarchy to ensure that each AUV maintains connectivity with higher level AUVs, thereby ensuring global connectivity. We summarize comprehensive evaluation criteria that can be utilized to select the optimal connection points, facilitating topology optimization. Simulation experiments demonstrate the impact of communication constraints on topology optimization and validate the effectiveness of the proposed method.