Issue tracking systems are used to manage and track software requirements and bug reports. In an issue tracking system, two bug reports created by two persons may be duplicate as they may refer to the same bug in a software system. Previous research shows that an estimated 35.8% to 41.6% of bug reports are duplicate, and this causes significant inefficiencies in software development and maintenance as developers have to waste time to identify if a bug report has duplicate reports prior to fixing the bug. Therefore, retrieving duplicate bug reports is important to improve productivity in software development. This paper proposes to use pre-trained language models (e.g., BERT and RoBERTa) to embed bug reports and to compute probability scores for retrieving duplicate bug reports. The proposed approach is evaluated with four common real datasets (i.e., Eclipse, Mozilla, NetBeans and Open Office) for the duplicate bug report retrieval task. The experimental results show that the proposed models are very effective for new duplicate bug report retrieval.