Autonomous valet parking system (AVP), which is based on trained trajectories achieving the functions of autonomous vehicle parking and shuttle within the designated map area, extends the scope of autonomous parking tasks. Accurate localization and the reconstruction of parking spaces are critical for the realization and stability of an AVP system. Traditional Visual SLAM (V-SLAM) often struggles with feature tracking loss in dynamic and crowded 3D parking spaces. Recently, robust road surface semantic features have been employed for AVP localization; however, sparse traffic markings and repetitive parking patterns slots challenges without precise initialization. In this paper, we introduce a hierarchical semantic map that incorporates road surface semantic markers and road topology semantic features (road category semantics tags) for use in autonomous valet parking. Compared to traditional V-SLAM or ground mark semantic SLAM, our system demonstrates robust performance in real-world scenarios, even in challenging lighting conditions and scenes with weak textures. We utilize an Around View Monitor (AVM) to construct the autonomous parking system, supplemented by an Inertial Measurement Unit (IMU) and wheel pulse encoder. The proposed system generates a global high-definition (HD) semantic map to ensure precise localization and robust collision avoidance. We analyze the recall and accuracy of our system’s collision-free space localization and compare it with other methods during real parking tasks. Furthermore, we showcase the strong capabilities of our proposed system in the application of autonomous valet parking.