Both the vanishing and the exploding gradient problem in feedforward neural networks can be prevented by using an appropriate weight initialization method together with carefully selected activation functions. This paper investigates 12 different configurations: Three weight initialization methods (random, Xavier, and He normal initialization) are combined with four different activation functions (ReLU, ELU, sigmoid, and tanh). These setting are evaluated by the average validation loss. The results indicate that for the selected Credit Card Dataset the choice of the activation function is more crucial than the selection of the weight initialization method. Furthermore, ReLU leads to the lowest loss value and sigmoid the highest regardless of which weight initialization method was used.