Scoliosis is the term for a curvature of the lumbar or thoracic spine in the coronal plane. Young children may develop spondylolisthesis and scoliosis, which, if left untreated, can grow into terrible agony. Lung and heart problems can also be caused by severe scoliosis. Early diagnosis can therefore help to arrest the disease’s progression and make it easier to apply therapies or interventions. Recent research has focused on the use of convolutional neural networks for the diagnosis of scoliosis and spondylolisthesis on X-ray images. Unfortunately, the majority of the current approaches ignore the larger-scale image contextual feature information in favour of gathering feature information for prediction from localised parts of images. Important features for classification—such as co-occurrence connections between labels and anatomical segmentation knowledge—are not completely utilised. This research suggests a Lightweight ResUNet (LW-RUnet) architecture that leverages the Xception backbone for feature extraction in order to segment scoliosis and spondylolisthesis using X-ray images. To fine-tune the disease area, the suggested architecture integrates the ResUNet decoder characteristics with the proposed middle decoder features. The outcomes of the suggested segmentation technique are verified using a number of measures, including accuracy, sensitivity, IOU, and dice similarity coefficient. With regard to the normal, scoliosis, and spondylolisthesis classes, the suggested method’s segmentation accuracy is 99.28%, 98.25, and 98.34, respectively. When compared with other deep learning models, the results demonstrate how well the suggested LW-RUNet technique performs.