Crystals are essential constituents of a wide range of materials, encompassing both advanced technologies and everyday applications. Recently, deep learning-based methods for crystal property prediction have demonstrated remarkable performance, greatly facilitating the discovery of novel materials. However, these approaches typically concentrate on atom-wise interactions and often fail to account for periodicity, a fundamental characteristic of crystals. To address this limitation, we propose a novel plug-and-play component, the Periodicity Injection Module (PIM), which seamlessly incorporates periodicity into existing crystal models. Specifically, the PIM employs crystal-wise attention to ensure that the surroundings of unit cells at periodic distances remain identical, aligning with the definition of crystal periodicity. By capturing interactions among bases at the crystal level, the PIM complements and enhances the modeling of intra-crystal interactions. Extensive experiments on benchmark datasets demonstrate that our PIM significantly improves crystal property prediction.