Quasi-maximum likelihood estimation and penalized estimation under non-standard conditions

The purpose of this article is to develop a general parametric estimation theory that allows the derivation of the limit distribution of estimators in non-regular models where the true parameter value may lie on the boundary of the parameter space or where even identifiability fails. For that, we propose a more general local approximation of the parameter space (at the true value) than previous studies. This estimation theory is comprehensive in that it can handle penalized estimation as well as quasi-maximum likelihood estimation (in the ergodic or non-ergodic statistics) under such non-regular models. In penalized estimation, depending on the boundary constraint, even the concave Bridge estimator does not necessarily give selection consistency. Therefore, we describe some sufficient condition for selection consistency, precisely evaluating the balance between the boundary constraint and the form of the penalty. An example is penalized MLE of variance components of random effects in linear mixed models.