On Modularity of Neural Networks: Systematic Review and Open Challenges

Modularity is used to manage the complexity of monolithic software systems and is a de facto practice in software engineering. Similar modularity concepts might also translate beneficially to machine learning, uncovering equivalent benefits, such as reuse opportunities. We address the research problem of modular neural networks’ (MNNs) applicability, operations, and comparability to monolithic solutions. A systematic literature review is used to identify 86 studies that provide information regarding modularity compared to monolithic solutions. The selected studies address many tasks and domains, evidencing broad applicability, although applied modularity operations are limited mainly to splitting. Nearly two-thirds of studies show improvements in task accuracy compared to monolithic solutions. Only 16% of studies report performance values in their comparisons, but 82% report MNN performance benefits in training and inference time, memory, and energy consumption compared to monolithic solutions. However, publication bias can favor MNNs, and most studies were conducted in laboratory environments on focused tasks and static requirements. Nevertheless, we conclude that MNNs perform at least satisfactorily compared to monolithic solutions. Modularity can bring forth benefits for managing complexity and has the potential for development and operations performance efficiency. Therefore, modularity in neural networks opens research avenues for extending software reuse, especially regarding broadening the applicability of advanced solutions and experiences from long-term or industrial applications and use.