Stochastic Uncertainty Quantification Techniques Fail to Account for Inter-analyst Variability in White Matter Hyperintensity Segmentation

White Matter Hyperintensities (WMH) are important neuroradiological markers of small vessel disease in brain MRI, with automatic segmentation tasks essential in research and clinical settings to understand their role in individuals’ health. However accurate segmentation of WMH is difficult due to their heterogeneous shape, intensity, size and location. Furthermore, image analysts working on different studies have adopted different approaches for providing accurate WMH segmentations, resulting in high inter-analyst variability. We assess the effectiveness of stochastic uncertainty quantification (UQ) techniques for bridging the variability in approaches and criteria in WMH segmentation. We first train six such techniques on an in-house dataset with two segmentation approaches, and then evaluate performance across three studies unseen by the model when training: Mild Stroke Study 3, the Lothian Birth Cohort 1936 and the WMH Challenge dataset. To aid in our analysis, we introduce two metrics: Uncertainty Inter Rater Overlap (UIRO) and Joint Uncertainty Error Overlap (JUEO). Our results show that changes in analyst policy between datasets dominates the uncertainty in the WMH segmentation task. Crucially, the distribution of segmentations predicted by stochastic models can fail to match the distribution of segmentations provided by analysts who are following approaches that differ from those used during training. We further suggest how to modify the task and cost function to overcome these difficulties.