With the continuous development of network encryption technologies, classifying encrypted traffic has become a focal point of recent research. Accurately classifying encrypted traffic remains challenging, yet it is crucial for enhancing network security and management. Existing methods that rely on pre-trained models often overlook the temporal characteristics of traffic data. We propose a model based on BERT and Recurrent Neural Networks for Encrypted Traffic Classification (BGETC) to address this issue. Our model operates in two phases: The first phase employs a BERT-based pre-training module designed to learn generic traffic representations from large-scale unlabelled encrypted traffic datasets. The second phase involves fine-tuning, where temporal features are incorporated at the output layer of the BERT model using Recurrent Neural Networks. Combining comprehensive features from the BERT model and temporal features significantly enhances performance, exceeding state-of-the-art baselines in several encrypted traffic classification tasks. Our method notably increases the F1 score to 99.39% for ISCX-VPN-App (packet) and 99.40% for USTC-TFC (flow), thereby markedly improving encrypted traffic classification performance.