The Generalized Neural Network (GNN) bundle combines the parallel processing capabilities of the High-Performance Computing Cluster (HPCC Systems) with the Keras and TensorFlow frameworks. It facilitates efficient development of neural network applications by leveraging the Enterprise Control Language (ECL), a robust Big Data processing language designed for distributed computing. In this work, we address performance challenges in distributed neural networks, suggesting that training on a single high-performance node (such as those with GPUs) or a limited number of nodes can yield better results in some cases. To improve performance, we optimized neural network training by capping the number of nodes, which minimized communication overhead and enhanced efficiency. Our research contributes to the ongoing development of the GNN bundle by demonstrating effective neural network training in distributed environments. We explore the nuances of parallelization and highlight the importance of selecting the right training approach based on the specific requirements of the neural network and the available computing resources. The findings offer valuable guidance for implementing efficient neural network training strategies in distributed computing contexts.