<p>The photon subtraction operation is proposed to enhance the entanglement between Gaussian entangled states, thereby improving the performance of continuous-variable quantum key distribution (CVQKD) systems. However, in practical CVQKD systems, the transmissivity of the beam splitters used in the photon subtraction operation may fluctuate, which may cause security issues. This paper investigates the practical security of CVQKD systems under imperfect photon subtraction operations and proposes security enhancement solutions. First, we model and analyze the impact of imperfections on the photon subtraction operation when the transmissivity of the beam splitters in the photon subtraction operation follows normal, beta, and exponential distributions respectively. These analyses show that such fluctuations reduce the success probability of the photon subtraction operation. Subsequently, if such fluctuations are not taken into account, as they may lead to an overestimation of the secret key rate due to inaccurate parameter estimation, they may introduce a security loophole into systems. To mitigate the negative impact of these fluctuations, we propose a new scheme, photon subtraction with the Mach–Zehnder interferometer (PSMZ), which is robust to fluctuations. In addition, we consider a Sagnac loop interferometric structure, where bidirectionally propagating modes travel along a common path and interfere at the same beam splitter. The proposed security improvement scheme can better mitigate transmissivity fluctuations, thereby improving the system performance and security.</p>