Most unsupervised anomaly detection methods learn a single decision boundary to distinguish samples in the training dataset, neglecting the variation for normal samples even in the same category and a distribution bias exists between the test set and the train set in the real world. Therefore, we propose an Adapted-MoE which contains a routing network and a series of expert models to handle multiple distributions of same-category samples by divide and conquer. Specifically, we propose a routing network to route same-category samples into the subclasses feature space. Then, a series of expert models are utilized to construct several independent decision boundaries. We propose the test-time adaption to eliminate the bias between the unseen test sample representation and the feature distribution learned by the expert model. Our experiments are conducted on a dataset that provides multiple subclasses from three categories, namely Texture AD. The Adapted-MoE significantly improves the performance of the baseline model, achieving 2.18%-7.20% and 1.57%-16.30% increase in I-AUROC and P-AUROC, which outperforms the current state-of-the-art methods. Our code is available at https://github.com/ray3572/AdaptedMoE .