The death and disability burden of spontaneous preterm births (sPTB) strikes millions of infants every year. Early risk stratification using transvaginal ultrasound imaging allows clinicians to implement preventive measures to delay or avoid preterm delivery. This makes sPTB prediction a highly valuable clinical target. Machine learning (ML) models have outperformed the clinical baseline—namely, cervical length (CL)—in numerous retrospective studies. Yet in practice, sPTB prevention still relies on the modest 40% sensitivity achieved by the standard CL threshold. A major barrier to clinical adoption of ML-based predictors is the lack of bias assessment. ML performance is known to vary across subpopulations, but studies rarely quantified where and for whom ML actually improves upon CL. Without subgroup-specific performance insights, clinicians are unable to identify the patients for whom ML would help—or harm—leading to justified reluctance toward its deployment. In our analysis, CL suffered its largest performance drops across physiological and patient-related parameters, whereas the deep learning model had a stronger sensitivity to imaging-related attributes. As a byproduct, we report a promising finding: Ultrasound images that display the cervix’s wide anatomical context appears to strongly boost deep learning performance.