Medical images often exhibit domain shifts owing to varying imaging protocols and scanners across different medical centres. To address this issue, Test-Time Adaptation (TTA) enables pre-trained models to adapt to test samples during inference. In this paper, we propose a novel method, termed Dynamic Normalization (DyNo), for medical image segmentation. Composed of two components, DyNo successfully alleviates domain shifts by adaptively mixing the statistics of multiple domains. We first demonstrated the feasibility of statistics-based methods which merge source and test statistics simply through a supervised toy experiment. Then, we introduce a synthetic domain that synthesizes the distribution information from both the source and target domains using moving average, thereby gradually bridging large domain shifts through the statistics of our synthetic domain. Next, we propose an adaptive fusion strategy, enabling our model to adapt to dynamically changing test data by estimating domain shifts in a fully hyperparameter-free manner. Our DyNo outperforms six competing TTA methods on two benchmark medical image segmentation tasks with multiple scenarios. Extensive ablation studies also demonstrate the effectiveness of synthetic statistics and our adaptive fusion strategy. The code and weights of pre-trained source models are available at https://github.com/Yihang-Fu/DyNo .