Single domain generalization aims to address the challenge of out-of-distribution generalization problem with only one source domain available. Feature disentanglement is a classic solution to this purpose, where the extracted task-related feature is presumed to be resilient to domain shift. However, the absence of references from other domains in a single-domain scenario poses significant uncertainty in feature disentanglement (ill-posedness). In this paper, we propose a new framework, named Domain Game, to perform feature disentangling for medical image segmentation, based on the observation that anatomical features are more sensitive to geometric transformations, whilst domain-specific features probably will remain invariant to such operations. Results from cross-site test domain evaluation showcase approximately an \(\sim \) 11.8% performance boost in prostate segmentation and around \(\sim \) 10.5% in brain tumor segmentation. The codes will be available at https://github.com/chqwer2/Domain-Game .