BERT has revolutionized the field of natural language processing through its exceptional performance on numerous tasks. Yet, the majority of researchers have mainly concentrated on enhancements related to the model structure and pretraining, such as relative position embedding, more efficient attention mechanisms, and Masked Language Modeling, including whole word masking. In this paper, we introduce BPDec (BERT Pretraining Decoder), a novel method for modeling pretraining. BPDec is enhanced with an addition of multiple transformer blocks as a decoder, the introduction of Gradual Unmasking Attention (GUA), and the integration of output randomness for a more dynamic final output. We utilize the original BERT model as the encoder so this approach does not necessitate extensive modifications to the encoder architecture and can be seamlessly integrated into existing fine-tuning pipelines and services, offering an efficient and effective enhancement strategy with no additional training cost during the fine-tuning phase. We test multiple enhanced decoder structures after pretraining and evaluate their performance on the GLUE tasks and SQuAD tasks. Our results demonstrate that BPDec, having only undergone subtle refinements to the model structure during pretraining, significantly enhances model performance without escalating the finetuning cost, inference time and serving budget.