Cloud platforms have become a primary environment for training machine learning models. However, the security of sensitive data could be compromised due to storage and computation in the cloud provided by the third party. Fully Homomorphic Encryption over the Torus (TFHE) offers a promising solution to this issue. However, integrating the wildly used Artificial Neural Networks (ANNs) with TFHE still faces significant challenges, particularly in the efficient implementation of activation functions and arithmetic operations. In this work, we present a parallel implementation of activation functions and arithmetic operations based on TFHE. By leveraging the parallel computing capabilities of multi-core CPUs and GPUs, our approach significantly improves computational performance. Experimental results demonstrate that our method achieves competitive accuracy while delivering a 12x speedup for 32-bit multiplication on GPUs and up to 30x acceleration for ReLU computations across various bit widths.