Maximum Likelihood Estimation

In this chapter, we will become familiar with a basic method for constructing estimators of unknown parameters in parametric models, which is the method of maximum likelihood (ML). This method, which has a long history, is applicable if the random variables on which the estimation is based have a density with respect to some dominating measure. The basic idea of ML estimation is to regard the parameter value that maximizes the joint density as a function of the parameter to be the most credible for the available data, and to take it as an estimate. Basic notions of this chapter are likelihood function, maximum likelihood estimator, Kullback-Leibler information, score vector, and Fisher information matrix. The first main result is the strong consistency of ML estimators under weak conditions on the density, provided that the parameter space is a compact set. The second main result is the asymptotic normality of the ML estimator under standard regularity conditions on the density, provided that the sequence of ML estimators is consistent.