Privacy-Preserving Machine Learning (PPML) has garnered considerable attention in recent years, with TFHE-based PPML playing a pivotal role in safeguarding user data privacy due to their accurate model inference capabilities. However, these schemes often rely on network quantization, and current research primarily focuses on model structure implementation and computational optimization, with limited emphasis on the design and implementation of activation functions. This study investigates the implementation of activation functions within the ciphertext domain of TFHE, specifically examining the performance of ReLU and its variants to enhance the practical feasibility of TFHE-based PPML applications. We developed a range of ciphertext-domain implementations for commonly used activation functions, offering plug-and-play interfaces that allow flexible activation function replacement in networks. Additionally, we simulated ciphertext inference using TFHE-based PPML to evaluate the performance of different activation functions and assess their computational time costs.