In recent years, homomorphic encryption (HE) has become a crucial tool for secure neural network inference (SNNI), which enables the server to classify encrypted data of clients while guaranteeing privacy. However, current HE-based frameworks limit the depth of neural networks. The main reason for the limitation is the noise and scaling factor growth in ciphertext after successive homomorphic operators. Gentry’s bootstrapping is normally the solution for addressing noise growth. However, bootstrapping is a costly procedure and requires the circular security assumption. For scaling factor growth, it remains a challenging problem because rescaling is based on division, which is not natively supported by current HE schemes. This paper proposes a double ciphertext refreshing protocol called DoubleR, which refreshes noise and scaling factor growth at the same time. Our protocol is proven secure in the semi-honest model without introducing additional assumptions. The experimental results show that our protocol outperforms bootstrapping by \(300 {\times }\) in running time. Based on DoubleR, we build a versatile framework for SNNI called HeSUN, which significantly accelerates the inference time with comparable communication costs.