Fetal growth restriction (FGR) is the main cause of neonatal morbidity and mortality. Previous research has attempted to identify FGR through ultrasound weight estimation and blood analysis; however, these approaches lack sufficient accuracy as they do not account for placental factors. Additionally, the intervention effect of these methods on FGR detection is limited, i.e., they can only be effective until the mid to late stages of pregnancy. In order to achieve accurate FGR detection in early pregnancy, this study makes full use of placental ultrasound images. A multi-scale learnable Bag-of-Visual-Words (BoVW) model that constructs visual words by learning to encode images is firstly proposed. Moreover, to make the proposed model pay attention to salient textures at different scales and positions, a spatial attention mechanism is incorporated. Experiments on ethically reviewed datasets validate the effectiveness of the proposed method in detecting FGR in early pregnancy.