In the field of wood veneer production, it is crucial to accurately and efficiently assess the image quality of wood veneers to improve the economic value. Existing assessment models based on convolutional neural networks (CNNs) are usually deployed with the structure of fixed-size convolutional kernels. However, these models have difficulty to extract multi-scale details and structures such as various defects and textures in wood veneer images. To improve the accuracy of wood veneer image quality assessment, this paper proposes a multi-scale attention hybrid network named WVIQA. The model is carried out in two steps. In the first step, we design a multi-scale feature extraction module which utilizes parallel convolutional layers containing different sized convolutional kernels to improve the model’s ability to extract multi-scale features. Another stage, we utilize the Transformer encoder’s global modeling capability to compensate for the limitation of a single CNN focusing on extracting local features. Moreover, we introduce Multi-scale Deformable Self Attention (MDSA) to replace the traditional self-attention mechanism in the Transformer encoder, which can dynamically adjust the attentional weights and is more in line with the pattern of the human visual system to focus on salient regions. We use the public datasets CLIVE, KonIQ-10k, and our own collected dataset Wood Veneer to demonstrate the effectiveness of WVIQA. On the Wood Veneer dataset, the PLCC metric reaches an impressive 89.1%, and our model outperforms other advanced models.