This study explores advanced techniques for optimizing the energy consumption of artificial and spiking neural networks, particularly in resource-constrained environments such as IoT devices. It presents a comprehensive overview of methods designed to reduce model size, computation time, and energy usage without significantly compromising accuracy. Among the studied approaches is the Local Zeroth-Order Optimization technique, which enables energy-efficient training of spiking neural networks despite the non-differentiability of the Heaviside function. This method achieves competitive generalization while significantly reducing the number of active neurons during backpropagation. Furthermore, the study investigates optimization strategies for artificial neural networks tailored to low-power microcontrollers like the ESP32 or nRF52840. These include pruning, quantization, and computational graph simplifications. Additionally, it presents a mixed-integer linear programming (MILP) model for workload distribution across edge, fog, and cloud layers, yielding energy savings of up to 86%. A novel application of passive optical networks is also introduced to reduce communication-related energy overhead. The study concludes with a discussion on the emerging potential of optical neural networks based on exciton-polariton technology. These systems offer significant performance and energy advantages over traditional CMOS-based architectures. The presented findings demonstrate the practical potential of combining neural network optimization with innovative hardware paradigms for future low-power AI applications.