Anomaly detection is an important task in time series analysis. The task of time series anomaly detection is often formulated as an unsupervised problem due to rare and costly annotation data. In many practical applications, the distribution of time series can be easily shifted, making it difficult to adapt the model to unseen data. In addition, in real-world applications, there is often a requirement to monitor and analyse lots of time series, and if we employ a separate model for each time series, the monitoring expense will significantly rise. Inspired by the pre-training models in natural language processing and computer vision, this paper proposes a representation learning framework for time series anomaly detection with temporal hierarchical and scalable multivariate fusion, which can process different multivariate time series flexibly and efficiently. This framework outperforms state-of-the-art anomaly detection algorithms in benchmark tests. In addition, the framework is used to train a pre-trained model on the mixed comprehensive datasets for the first time. The performance of the pre-trained model is comparable to models trained on a single dataset. Furthermore, the pre-trained model achieves excellent results on an unseen dataset, with no fine-tuning or after fine-tuning by a few examples.