Effective fusion of acoustic spatial cues from echo signals and informative visual cues from images is essential for accurate multimodal depth estimation, as each modality provides complementary information while also exhibiting inherent limitations. To achieve better fusion of multimodal data, we introduce HM-Net, a hierarchical multimodal depth estimation framework that integrates echo and visual information through both low level feature fusion and high level semantic fusion. During the low level feature fusion stage, we employ two attention-weighted multimodal fusion modules, with one enhancing echo features under visual guidance and the other enhancing visual features under echo guidance. Each module adjusts modality-specific weights based on cross-modal interactions, resulting in adaptive multimodal features. During the high level semantic fusion stage, these multimodal features are processed to generate scene intervals that represent the global structural layout of the scene, scene feature maps that serve as semantic refinements to the global structure delineated by the scene intervals, capturing localized depth cues such as object boundaries, surface continuity, and occlusion relationships. By fusing these two high level semantic representations, we predict the final depth map, which captures the global depth distribution of the scene and enables precise depth estimation. Quantitative and qualitative evaluations on the Replica, Matterport3D, and BatVision datasets demonstrate the exceptional performance of our method in multimodal depth estimation tasks. The source code is available at: https://github.com/henu77/HM_Net .