With a huge growth of artificial intelligence, deep learning neural network has been started to be used in various fields. Despite an inference process of the neural network requires important parameters, such as trained weight, bias, or private input data of users, the neural network was not designed with consideration of side-channel vulnerability and the risk of information leakage through side-channel attack has became a security threat to the neural network. For this reason, many prior studies presented side-channel countermeasures (e.g. masking or shuffling) for neural network. However, the state-of-the-art masking methods has been designed only for ReLU (Rectified Linear Unit), and cannot covers other activation functions using exponential parts. Thus, we will propose an idea to apply masking method on exponential activation functions, using the strategy of replacement. Replacing exponential function with simpler operation not only enables to extend the target of masking countermeasure in AI field, but also gives benefit in the aspect of speed. We will present two replacement scenarios, the polynomial approximation and Table implementation, with design of masked algorithms to perform replaced activation function and the result of performance and security evaluations about masked neural networks.