In the field of computer vision, traditional Vision Transformer (ViT) is usually weak in capturing local features due to the lack of inductive bias. Meanwhile, in the embedding layer we directly use CNN to convert the input image into a series of tokens, but the localization of CNN doesn’t adequately preserve some of the original image information, which leads to poor performance of the network training. To address these two problems, this paper proposes a new visual transformer network (ADViT) based on adaptive feature selection and global deformable convolution. First, we designed an adaptive learning module based on k-nearest neighbor (kNN), aiming to mitigate the impact of missing inductive bias in traditional Transformer. Secondly, we integrate a globally deformable convolution module to enhance the fusion of local features and global features and improve the ability to capture image details. Our extensive experiments on the NYUD-v2 dataset show that ADViT achieves a 3.7% improvement in mIoU on the image segmentation task compared to the existing optimal method. ADViT not only improves the accuracy of the model, but also adds an edge in computational efficiency.