Texture recognition remains a challenging visual task due to the complex appearance variants caused by scale changes in real world. Features from different scales of objects may exhibit similarity, leading to confusion when classifying the texture of objects. However, most existing texture recognition approaches based on convolutional neural networks typically represent textures at a single scale, which limits their ability to leverage the complementary relationships among multi-scales features and thus constrains their performance. To address this issue, we propose a dilated pyramid attention (DPA) module, in which a group of dilated convolutions with gradually increased dilation ratios are applied to extract more multi-scales features and an attention mechanism is used to calculate the correlation among them for more accurate recognition. Furthermore, we propose a hierarchical vision transformer, named DPA-ViT, which integrates the DPA module to fully exploit multi-scales features for texture recognition. Extensive experiments demonstrate the effectiveness of our methods. Our DPA-ViT achieve state-of-the-art on GTOS-Mobile and comparable results on other datasets.