An iterative two-step regularization approach for selection of fixed and random effects in generalized linear mixed models

The generalized linear mixed model (GLMM) is one of the most widely used methods to account for possible association among clustered and typically non-normal data arising from longitudinal or multi-center studies. Due to non-linear link function in the model and possibly high-dimensional integration in the marginal likelihood, it brings challenges in the selection of fixed and random effects simultaneously. In this paper, we propose a novel regularization approach for identifying both important fixed and random effects in GLMMs with high-dimensional sparse effects. The proposed approach utilizes approximate inference to connect the estimation problem in GLMMs to that in the framework of linear mixed models. It then employs a two-step procedure to iteratively estimate and identify fixed effects and random effects, using a penalized profile likelihood and a penalized restricted posterior likelihood, respectively. Our proposal is particularly appealing as it regularizes either the fixed or random effects only in each step, leading to a more efficient and stable computation than the joint selection for both types of effects. We demonstrate its favorable finite sample performance through extensive simulation studies and illustrate its application using three real data examples with different types of responses in longitudinal or clustered settings.