Visual geo-localization is the ability to recognize the geographical location of a query image through visual clues in stored geo-tagged images. It is critical for applications like simultaneous localization and mapping (SLAM) systems in robotics and self-driving vehicles. However, in low-perspective urban street imagery, non-landmark elements, such as dynamic objects, including riders, pedestrians and vehicles, and geographically insignificant regions, like the sky, grass and trees, may introduce noise information and affect the accuracy of geo-localization. In this paper, we propose SemVG, a semantic fused feature extraction network to obtain robust and compact visual descriptors for urban street imagery thus to improve geo-localization accuracy. SemVG utilizes the visual foundation model Dinov2 ViT as its backbone and adds a feature selection and fusion process. First, we use a pre-trained semantic segmentation network to generate a semantic mask at the patch-level where the patch size is aligned with local features extracted from the Dinov2 ViT model. The semantic mask is applied to filter out local features for non-landmark image patches and classify the remains into groups based on their semantic categories. These local feature groups are then processed into more compact semantic-level features, which are aggregated and merged with the global feature to produce the final descriptor. The network is finetuned on a large urban dataset with a Multi-Similarity loss function in a metric learning manner. Extensive evaluations demonstrate that SemVG achieves new state-of-the-art performance on several urban visual geo-localization benchmarks by excluding non-landmark information and integrating semantic information into final features.