<p>Code completion is a crucial aspect of contemporary integrated development environments (IDEs), as it not only streamlines the software development process but also bolsters the quality of software products. By leveraging large-scale codes to learn the probability distribution among code token units, deep learning methods have demonstrated significant improvements in the accuracy of token unit recommendations. However, the efficacy of code completion employing deep learning is often compromised by information loss. To mitigate this issue, we introduce a novel code language model that incorporates both the pointer network and the Transformer-XL architecture to surpass the constraints of current approaches in code completion. Our proposed model accepts as input the original code snippet and its corresponding abstract syntax tree (AST), utilizing the Transformer-XL model as the foundational architecture for capturing long-term dependencies. Additionally, we incorporate a pointer network as an adjunct component to forecast Out-of-Vocabulary (OoV) words. Our approach has been rigorously evaluated on the authentic PY150 and JS150 datasets. The comparative experimental results demonstrate the effectiveness of our model in improving the accuracy of the code completion task at the token unit level.</p>