Neural Architecture Search (NAS) methods are used to automate the design of neural network architectures. On many popular datasets, architectures discovered through NAS already outperform manually designed ones. However, most existing approaches rely on predefined sets of common operations, such as fully connected or convolutional layers with a fixed set of their parameters. As a result, these methods are limited in their ability to discover fundamentally new architectures with complex topologies. In this work, we propose an approach that designs architectures at the most fundamental level—by directly manipulating neurons and their connections. In addition, we focus on finding computationally efficient models by eliminating redundant components that may be inherent in manually constructed layers. To achieve this, we introduce an evolutionary algorithm that balances computational cost with architecture performance. Experimental results demonstrate that our approach can discover architectures that match the performance of pruned, manually designed networks while achieving approximately a twofold reduction in FLOPs on image classification datasets. Additionally, the use of specialized tools for sparse model acceleration allows our architectures to achieve up to a 20% reduction in inference time on the CPU compared to manually designed networks.