Artificial neural networks have become crucial across fields like IoT, computer vision, and medicine. Their use for a variety of industrial applications, and the ever-expanding IoT, contributed to a growing interest for lightweight neural network models suitable for deployment in environments with limited computational capabilities. Pruning techniques aim to address the computational and storage demands of these models. In this work, we investigate the impact that different pruning methods have on Multi-Layer Perceptron (MLP) networks. We compare pre-training, in-training, post-training, and the SET-Method pruning approaches while considering a variety of parameters. We find that highly sparse small-scale MLPs can achieve accuracies similar to their fully connected counterparts. Furthermore, energy consumption and inference time are primarily influenced by model size, rather than sparsity levels. This research provides insight into optimizing and further understanding neural networks and their applicability to real-world applications.