Scoliosis is a spinal disorder characterised by a lateral curvature of the spine, typically diagnosed using X-ray imaging. In this paper we propose a modality-agnostic method to predict phenotypes of scoliosis. These phenotypes describe the curve pattern, and include the number of (significant) curves, the location and direction of the largest curve, as well as whether the spine in general is scoliotic or not. The method is modality-agnostic in the sense that it can be applied to multiple imaging modalities. The method involves representing the spine curve using the coefficients of a low-dimensional Fourier sine series expansion, and then obtaining the phenotypes from these coefficients using a simple feed-forward neural network. The network is trained on curves extracted from DXA images, but can then be applied ‘as-is’ (without fine-tuning) to curves extracted from MRIs and X-rays. We evaluate the performance of the method on three datasets, one for each modality, and demonstrate excellent performance.