The automated segmentation of brain structures, such as the hippocampus and basal ganglia, from ultra-low field (0.064T) neonatal magnetic resonance imaging (MRI) is important for clinical diagnosis and neurodevelopmental research. Manual segmentation is time-consuming and subject to inter- and intra-observer variability. The unique challenges of ultra-low field MRI, including low signal to noise ratio and low spatial resolution make automated segmentation a difficult task. In this study, the challenge of automated segmentation of the basal ganglia and hippocampus in pediatric ultra-low field MRI is addressed. The approach builds on MedNeXt, a transformer-inspired, fully convolutional encoder-decoder architecture, trained with the nnU-Net pipeline. To address the class imbalance inherent in segmenting small structures, a combined Focal-Dice-CrossEntropy loss function was employed. The method was evaluated using the Dice Similarity Coefficient (DSC), 95th Percentile Hausdorff Distance (HD95), Average Symmetric Surface Distance (ASSD), and Relative Volume Error (RVE). The results show an average DSC of 0.69 ± 0.18 for hippocampal segmentation and an average DSC of 0.85 ± 0.06 for basal ganglia segmentation. The method demonstrated better performances in segmenting the basal ganglia in the ultra-low field images as compared to the hippocampus.