Deep learning models deliver impressive accuracy but usually demand large memory and compute resources, which makes them difficult to use in mobile or embedded environments. In earlier work we proposed the Distribution Based Neural Network (DBNN), a compact architecture that encodes connection weights through Gaussian distributions and quantile breakpoints rather than storing full dense matrices. In this paper, we take a closer look at DBNN and examine how well it performs under different conditions. Unlike Bayesian approaches that only sample weights, or Hypernetworks that rely on a separate network to generate them, DBNN learns a small set of distribution parameters that can reproduce full weight matrices, keeping the model lightweight while still effective. Training is carried out with a Genetic Algorithm, well suited to the sparse parameter space where gradient descent tends to fail. We study DBNN through ablation experiments on activation and fitness functions, and through robustness tests that vary hidden nodes and input dimensionality. The final ensemble model is evaluated on 104 binary classification datasets from OpenML against GANN, Hypernetworks, MLPs, and four classical machine learning baselines. DBNN reaches an average AUC of 0.854, outperforming GANN and approaching Hypernetworks and MLPs, while requiring only 2–11% of their parameters. This makes DBNN a promising candidate resource constrained settings like IoT, edgeAI and mobile computing.