Depth maps captured by low-cost depth cameras are usually of low resolution, which heavily restricts the application of depth information in various computer vision tasks. Color-guided depth map super-resolution (CDSR) methods have made rapid advancements in addressing this issue. However, existing CDSR methods often suffer from unexpected artifacts and unsatisfactory results, especially around the edges of depth maps, due to the discrepancies between color and depth maps. To overcome this problem, we propose a novel method for color-guided depth map super-resolution called EAY-Net. The proposed EAY-Net is a Y-shaped network that includes two separate encoder branches for processing color and depth maps, a decoder branch for feature reconstruction, and multi-scale skip connections (MSC) to effectively combine features from both encoder branches and the decoder branch, ensuring comprehensive feature integration. To facilitate the extraction of color features tailored to the paired depth map, we propose integrating an adaptive image feature extraction module (AFEM). This integration significantly contributes to the mitigation of artifacts, further enhancing the quality of the results. Extensive experiments demonstrate that our proposed method outperforms state-of-the-art methods.