<p>Many real-world machine learning applications are characterized by a huge number of features, leading to computational and memory issues, as well as the risk of overfitting. Ideally, only relevant and non-redundant features should be considered to preserve the complete information of the data and limit the dimensionality. Dimensionality reduction and feature selection are common preprocessing techniques addressing the challenge of efficiently dealing with high-dimensional data. Dimensionality reduction methods control the number of features in a dataset while minimizing information loss. Feature selection aims to identify the most relevant features for a task, discarding the less informative ones. Previous works have proposed approaches that aggregate features depending on their correlation without discarding any of them and preserving their interpretability through aggregation with the mean. A limitation of these works is the assumption of linearity in the relationship between features and targets. In this paper, we relax this assumption in two ways. First, we propose a bias-variance analysis for general regression models with additive Gaussian noise, leading to a first dimensionality reduction algorithm (NonLinCFA). Then, we extend the approach assuming that a generalized (non-)linear model regulates the data generation process. A deviance analysis leads to a second dimensionality reduction algorithm (GenLinCFA), applicable to larger classes of regression problems and in classification. In both cases, the main focus is to preserve the interpretability of the reduced features through the aggregation with the mean of groups of original features. Finally, we test the algorithms on synthetic and real-world datasets, performing regression and classification and showing competitive performance.</p>