Maximum likelihood estimation

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of a statistical model given observations, by finding the parameter values that maximize the likelihood of making the observations given the parameters. MLE can be seen as a special case of the maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters, or as a variant of the MAP that ignores the prior and which therefore is unregularized.

The method of maximum likelihood corresponds to many well-known estimation methods in statistics. For example, one may be interested in the heights of adult female penguins, but be unable to measure the height of every single penguin in a population due to cost or time constraints. Assuming that the heights are normally distributed with some unknown mean and variance, the mean and variance can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE would accomplish this by taking the mean and variance as parameters and finding particular parametric values that make the observed results the most probable given the model.

In general, for a fixed set of data and underlying statistical model, the method of maximum likelihood selects the set of values of the model parameters that maximizes the likelihood function. Intuitively, this maximizes the "agreement" of the selected model with the observed data, and for discrete random variables it indeed maximizes the probability of the observed data under the resulting distribution. Maximum likelihood estimation gives a unified approach to estimation, which is well-defined in the case of the normal distribution and many other problems.

History

Ronald Fisher in 1913

Maximum-likelihood estimation was recommended, analyzed (with fruitless attempts at proofs) and widely popularized by Ronald Fisher between 1912 and 1922^[1] (although it had been used earlier by Carl Friedrich Gauss, Pierre-Simon Laplace, Thorvald N. Thiele, and Francis Ysidro Edgeworth).^[2]

Maximum-likelihood estimation finally transcended heuristic justification in a proof published by Samuel S. Wilks in 1938, now called "Wilks' theorem".^[3] The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent samples is χ² distributed, which enables determination of a confidence region around any one estimate of the parameters. Ironically, the only difficult part of the proof depends on the expected value of the Fisher information matrix, which is provided by a theorem by Fisher.^[4] Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.^[5]

Some of the theory behind maximum likelihood estimation was developed for Bayesian statistics.^[1]

Reviews of the development of maximum likelihood estimation have been provided by a number of authors.^[6]

Principles

Suppose there is a sample x₁, x₂, …, x_n of n independent and identically distributed observations, coming from a distribution with an unknown probability density function f₀(·). It is however surmised that the function f₀ belongs to a certain family of distributions { f(·| θ), θ ∈ Θ } (where θ is a vector of parameters for this family), called the parametric model, so that f₀ = f(·| θ₀). The value θ₀ is unknown and is referred to as the true value of the parameter vector. It is desirable to find an estimator $\scriptstyle {\hat {\theta }}$ which would be as close to the true value θ₀ as possible. Either or both the observed variables x_i and the parameter θ can be vectors.

To use the method of maximum likelihood,^[7] one first specifies the joint density function for all observations. For an independent and identically distributed sample, this joint density function is

f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )=f(x_{1}\mid \theta )\times f(x_{2}\mid \theta )\times \cdots \times f(x_{n}\mid \theta ).

Now we look at this function from a different perspective by considering the observed values x₁, x₂, …, x_n to be fixed "parameters" of this function, whereas θ will be the function's variable and allowed to vary freely; this same function will be called the likelihood:

{\mathcal {L}}(\theta \,;\,x_{1},\ldots ,x_{n})=f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )=\prod _{i=1}^{n}f(x_{i}\mid \theta ).

Note that " $;$ " denotes a separation between the two categories of input arguments: the parameters $\theta$ and the observations $x_{1},\ldots ,x_{n}$ .

In practice it is often more convenient when working with the natural logarithm of the likelihood function, called the log-likelihood:

\ln {\mathcal {L}}(\theta \,;\,x_{1},\ldots ,x_{n})=\sum _{i=1}^{n}\ln f(x_{i}\mid \theta ),

or the average log-likelihood:

{\hat {\ell }}={\frac {1}{n}}\ln {\mathcal {L}}.

The hat over ℓ indicates that it is akin to some estimator. Indeed, $\scriptstyle {\hat {\ell }}$ estimates the expected log-likelihood of a single observation in the model.

The method of maximum likelihood estimates θ₀ by finding a value of θ that maximizes ${\hat {\ell }}(\theta ;x)$ . This method of estimation defines a maximum likelihood estimator (MLE) of θ₀:

\{{\hat {\theta }}_{\mathrm {mle} }\}\subseteq \{{\underset {\theta \in \Theta }{\operatorname {arg\,max} }}\ {\hat {\ell }}(\theta \,;\,x_{1},\ldots ,x_{n})\},

if a maximum exists. An MLE estimate is the same regardless of whether we maximize the likelihood or the log-likelihood function, since log is a monotonically increasing function.

For many models, a maximum likelihood estimator can be found as an explicit function of the observed data x₁, ..., x_n. For many other models, however, no closed-form solution to the maximization problem is known or available, and an MLE has to be found numerically using optimization methods. For some problems, there may be multiple estimates that maximize the likelihood. For other problems, no maximum likelihood estimate exists – either the log-likelihood function increases without ever reaching a supremum value, or the supremum does exist but is outside the bounds of $\Theta$ , the set of acceptable parameter values.

In the exposition above, it is assumed that the data are independent and identically distributed. The method can be applied however to a broader setting, as long as it is possible to write the joint density function f(x₁, …, x_n | θ), and its parameter θ has a finite dimension which does not depend on the sample size n. In a simpler extension, an allowance can be made for data heterogeneity, so that the joint density is equal to f₁(x₁ | θ) · f₂(x₂|θ) · ··· · f_n(x_n | θ). Put another way, we are now assuming that each observation x_i comes from a random variable that has its own distribution function f_i. In the more complicated case of time series models, the independence assumption may have to be dropped as well.

A maximum likelihood estimator coincides with the most probable Bayesian estimator given a uniform prior distribution on the parameters. Indeed, the maximum a posteriori estimate is the parameter θ that maximizes the probability of θ given the data, given by Bayes' theorem:

P(\theta \mid x_{1},x_{2},\ldots ,x_{n})={\frac {f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )P(\theta )}{P(x_{1},x_{2},\ldots ,x_{n})}}

where $P(\theta )$ is the prior distribution for the parameter θ and where $P(x_{1},x_{2},\ldots ,x_{n})$ is the probability of the data averaged over all parameters. Since the denominator is independent of θ, the Bayesian estimator is obtained by maximizing $f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )P(\theta )$ with respect to θ. If we further assume that the prior $P(\theta )$ is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function $f(x_{1},x_{2},\ldots ,x_{n}\mid \theta )$ . Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution $P(\theta )$ .

Properties

A maximum likelihood estimator is an extremum estimator obtained by maximizing, as a function of θ, the objective function (c.f., the loss function)

{\hat {\ell }}(\theta \,;\,x)={\frac {1}{n}}\sum _{i=1}^{n}\ln f(x_{i}\mid \theta ),

this being the sample analogue of the expected log-likelihood $\ell (\theta )=\operatorname {E} [\,\ln f(x_{i}\mid \theta )\,]$ , where this expectation is taken with respect to the true density $f(\cdot \mid \theta _{0})$ .

Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.^[8] However, like other estimation methods, maximum likelihood estimation possesses a number of attractive limiting properties: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties:

Consistency: the sequence of MLEs converges in probability to the value being estimated.
Asymptotic normality: as the sample size increases, the distribution of the MLE tends to the Gaussian distribution with mean $\theta$ and covariance matrix equal to the inverse of the Fisher information matrix.
Efficiency, i.e., it achieves the Cramér–Rao lower bound when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic mean squared error than the MLE (or other estimators attaining this bound).
Second-order efficiency after correction for bias.

Consistency

Under the conditions outlined below, the maximum likelihood estimator is consistent. The consistency means that having a sufficiently large number of observations n, it is possible to find the value of θ₀ with arbitrary precision. In mathematical terms this means that as n goes to infinity the estimator $\scriptstyle {\hat {\theta }}$ converges in probability to its true value:

${\begin{matrix}{}\\{\hat {\theta }}_{\mathrm {mle} }\ {\xrightarrow {p}}\ \theta _{0}.\\{}\end{matrix}}$

(1)

Under slightly stronger conditions, the estimator converges almost surely (or strongly) to:

${\begin{matrix}{}\\{\hat {\theta }}_{\mathrm {mle} }\ {\xrightarrow {\text{a.s.}}}\ \theta _{0}.\\{}\end{matrix}}$

(2)

To establish consistency, the following conditions are sufficient:^[9]

Identification of the model:
$\theta \neq \theta _{0}\quad \Leftrightarrow \quad f(\cdot \mid \theta )\neq f(\cdot \mid \theta _{0}).$

In other words, different parameter values θ correspond to different distributions within the model. If this condition did not hold, there would be some value θ₁ such that θ₀ and θ₁ generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have been observationally equivalent.
The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function ℓ(θ|·) has unique global maximum at θ₀.
Compactness: the parameter space Θ of the model is compact.

The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).

Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:
- both concavity of the log-likelihood function and compactness of some (nonempty) upper level sets of the log-likelihood function, or
- existence of a compact neighborhood N of θ₀ such that outside of N the log-likelihood function is less than the maximum by at least some ε > 0.
Continuity: the function ln f(x|θ) is continuous in θ for almost all values of x:
$\Pr \!{\big [}\;\ln f(x\mid \theta )\;\in \;\mathbb {C} ^{0}(\Theta )\;{\big ]}=1.$
The continuity here can be replaced with a slightly weaker condition of upper semi-continuity.
Dominance: there exists D(x) integrable with respect to the distribution f(x|θ₀) such that
${\big |}\ln f(x\mid \theta ){\big |}<D(x)\quad {\text{ for all }}\theta \in \Theta .$

By the uniform law of large numbers, the dominance condition together with continuity establish the uniform convergence in probability of the log-likelihood:

$\sup _{\theta \in \Theta }{\big |}{\hat {\ell }}(\theta \mid x)-\ell (\theta )\,{\big |}\ {\xrightarrow {p}}\ 0.$

The dominance condition can be employed in the case of i.i.d. observations. In the non-i.i.d. case the uniform convergence in probability can be checked by showing that the sequence $\scriptstyle {\hat {\ell }}(\theta \mid x)$ is stochastically equicontinuous.

If one wants to demonstrate that the ML estimator $\scriptstyle {\hat {\theta }}$ converges to θ₀ almost surely, then a stronger condition of uniform convergence almost surely has to be imposed:

\sup _{\theta \in \Theta }{\big \|}\;{\hat {\ell }}(x\mid \theta )-\ell (\theta )\;{\big \|}\ {\xrightarrow {\text{a.s.}}}\ 0.

Asymptotic normality

In a wide range of situations, maximum likelihood parameter estimates exhibit asymptotic normality – that is, they are equal to the true parameters plus a random error that is approximately normal (given sufficient data), and the error's variance decays as 1/n. For this property to hold, it is necessary that the estimator does not suffer from the following issues:

Estimate on boundary

Sometimes the maximum likelihood estimate lies on the boundary of the set of possible parameters, or (if the boundary is not, strictly speaking, allowed) the likelihood gets larger and larger as the parameter approaches the boundary. Standard asymptotic theory needs the assumption that the true parameter value lies away from the boundary. If we have enough data, the maximum likelihood estimate will keep away from the boundary too. But with smaller samples, the estimate can lie on the boundary. In such cases, the asymptotic theory clearly does not give a practically useful approximation. Examples here would be variance-component models, where each component of variance, $σ 2$ , must satisfy the constraint $σ 2 \geq 0$ .

Data boundary parameter-dependent

For the theory to apply in a simple way, the set of data values which has positive probability (or positive probability density) should not depend on the unknown parameter. A simple example where such parameter-dependence does hold is the case of estimating θ from a set of independent identically distributed observations when the common distribution is uniform on the range (0,θ). For estimation purposes the relevant range of θ is such that θ cannot be less than the largest observation. Because the interval (0,θ) is not compact, there exists no maximum for the likelihood function: For any estimate of theta, there exists a greater estimate that also has greater likelihood. In contrast, the interval [0,θ] includes the end-point θ and is compact, in which case the maximum likelihood estimator exists. However, in this case, the maximum likelihood estimator is biased. Asymptotically, this maximum likelihood estimator is not normally distributed.^[10]

Nuisance parameters

For maximum likelihood estimations, a model may have a number of nuisance parameters. For the asymptotic behaviour outlined to hold, the number of nuisance parameters should not increase with the number of observations (the sample size). A well-known example of this case is where observations occur as pairs, where the observations in each pair have a different (unknown) mean but otherwise the observations are independent and normally distributed with a common variance. Here for 2N observations, there are N + 1 parameters. It is well known that the maximum likelihood estimate for the variance does not converge to the true value of the variance.

Increasing information

For the asymptotics to hold in cases where the assumption of independent identically distributed observations does not hold, a basic requirement is that the amount of information in the data increases indefinitely as the sample size increases. Such a requirement may not be met if either there is too much dependence in the data (for example, if new observations are essentially identical to existing observations), or if new independent observations are subject to an increasing observation error.

Some regularity conditions which ensure this behavior are:

The first and second derivatives of the log-likelihood function exist (are “well defined”).
The Fisher information matrix is non-singular.
The Fisher information matrix is continuous as a function of the parameters, θ.
The maximum likelihood estimator is consistent.

Suppose that conditions for consistency of maximum likelihood estimator are satisfied, and^[11]

$θ 0 \in interior(Θ)$ ;
$f (x | θ) > 0$ and is twice continuously differentiable in $Θ$ in some neighborhood N of $θ 0$ ;
$\int sup θ \in N ||\nabla θ f (x | θ)||d x < \infty, and \int sup θ \in N ||\nabla θθ f (x | θ)||d x < \infty$ ;
$I = E[\nabla θ ln f (x | θ 0) \nabla θ ln f (x | θ 0)']$ exists and is nonsingular;
$E[sup θ \in N ||\nabla θθ ln f (x | θ)||] < \infty$ .

Then the maximum likelihood estimator has asymptotically normal distribution:

{\sqrt {n}}{\big (}{\hat {\theta }}_{\mathrm {mle} }-\theta _{0}{\big )}\ {\xrightarrow {d}}\ {\mathcal {N}}(0,\,I^{-1}).

Sketch of proof

Since the log-likelihood function is differentiable, and $\theta _{0}$ lies in the interior of the parameter set $\Theta$ , in the maximum the first-order condition will be satisfied:

\nabla _{\!\theta }\,{\hat {\ell }}({\hat {\theta }}\mid x)={\frac {1}{n}}\sum _{i=1}^{n}\nabla _{\!\theta }\ln f(x_{i}\mid {\hat {\theta }})=0.

When the log-likelihood is twice differentiable, this expression can be expanded into a Taylor series around the point $\theta =\theta _{0}$ :

0={\frac {1}{n}}\sum _{i=1}^{n}\nabla _{\!\theta }\ln f(x_{i}\mid \theta _{0})+{\Bigg [}\,{\frac {1}{n}}\sum _{i=1}^{n}\nabla _{\!\theta \theta }\ln f(x_{i}\mid {\tilde {\theta }})\,{\Bigg ]}({\hat {\theta }}-\theta _{0}),

where ${\tilde {\theta }}$ is some point intermediate between $\theta _{0}$ and $\hat{\theta}$ . From this expression we can derive that

{\sqrt {n}}({\hat {\theta }}-\theta _{0})={\Bigg [}\,{-{\frac {1}{n}}\sum _{i=1}^{n}\nabla _{\!\theta \theta }\ln f(x_{i}\mid {\tilde {\theta }})}\,{\Bigg ]}^{-1}{\frac {1}{\sqrt {n}}}\sum _{i=1}^{n}\nabla _{\!\theta }\ln f(x_{i}\mid \theta _{0})

Here the expression in square brackets converges in probability to $H=\mathbb {E} \left[-\nabla _{\theta \theta }\ln f(x|\theta _{0})\right]$ by the law of large numbers. The continuous mapping theorem ensures that the inverse of this expression also converges in probability, to $H^{{-1}}$ . The second sum, by the central limit theorem, converges in distribution to a multivariate normal with mean zero and variance matrix equal to the Fisher information $I$ . Thus, applying Slutsky's theorem to the whole expression, we obtain that

{\sqrt {n}}({\hat {\theta }}-\theta _{0})\ \ {\xrightarrow {d}}\ \ {\mathcal {N}}{\big (}0,\ H^{-1}IH^{-1}{\big )}.

Finally, the information equality guarantees that when the model is correctly specified, matrix $H$ will be equal to the Fisher information $I$ , so that the variance expression simplifies to just $I^{-1}$ .

Functional invariance

The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if ${\widehat {\theta }}$ is the MLE for θ, and if g(θ) is any transformation of θ, then the MLE for α = g(θ) is by definition

{\widehat {\alpha }}=g(\,{\widehat {\theta }}\,).\,

It maximizes the so-called profile likelihood:

{\bar {L}}(\alpha )=\sup _{\theta :\alpha =g(\theta )}L(\theta ).\,

The MLE is also invariant with respect to certain transformations of the data. If Y = g(X) where g is one to one and does not depend on the parameters to be estimated, then the density functions satisfy

f_{Y}(y)={\frac {f_{X}(x)}{|g'(x)|}}

and hence the likelihood functions for X and Y differ only by a factor that does not depend on the model parameters.

For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data.

Higher-order properties

The standard asymptotics tells that the maximum likelihood estimator is √n-consistent and asymptotically efficient, meaning that it reaches the Cramér–Rao bound:

{\sqrt {n}}({\hat {\theta }}_{\text{mle}}-\theta _{0})\ \ {\xrightarrow {d}}\ \ {\mathcal {N}}(0,\ I^{-1}),

where I is the Fisher information matrix:

I_{jk}=\operatorname {E} _{X}{\bigg [}\;{-{\frac {\partial ^{2}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{j}\,\partial \theta _{k}}}}\;{\bigg ]}.

In particular, it means that the bias of the maximum likelihood estimator is equal to zero up to the order n^−1/2. However, when we consider the higher-order terms in the expansion of the distribution of this estimator, it turns out that θ_mle has bias of order n⁻¹. This bias is equal to (componentwise)^[12]

b_{s}\equiv \operatorname {E} [({\hat {\theta }}_{\mathrm {mle} }-\theta _{0})_{s}]={\frac {1}{n}}\cdot I^{si}I^{jk}{\big (}{\tfrac {1}{2}}K_{ijk}+J_{j,ik}{\big )}

where Einstein's summation convention over the repeating indices has been adopted; I^jk denotes the j,k-th component of the inverse Fisher information matrix I⁻¹, and

{\tfrac {1}{2}}K_{ijk}+J_{j,ik}=\operatorname {E} _{X}{\bigg [}\;{\frac {1}{2}}{\frac {\partial ^{3}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{i}\,\partial \theta _{j}\,\partial \theta _{k}}}+{\frac {\partial \ln f_{\theta _{0}}(X_{t})}{\partial \theta _{j}}}{\frac {\partial ^{2}\ln f_{\theta _{0}}(X_{t})}{\partial \theta _{i}\,\partial \theta _{k}}}\;{\bigg ]}.

Using these formulas it is possible to estimate the second-order bias of the maximum likelihood estimator, and correct for that bias by subtracting it:

{\hat {\theta }}_{\mathrm {mle} }^{*}={\hat {\theta }}_{\mathrm {mle} }-{\hat {b}}.

This estimator is unbiased up to the terms of order n⁻¹, and is called the bias-corrected maximum likelihood estimator.

This bias-corrected estimator is second-order efficient (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order n⁻². It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, as was shown by Kano (1996), the maximum likelihood estimator is not third-order efficient.

Examples

Discrete uniform distribution

Consider a case where n tickets numbered from 1 to n are placed in a box and one is selected at random (see uniform distribution); thus, the sample size is 1. If n is unknown, then the maximum likelihood estimator ${\hat {n}}$ of n is the number m on the drawn ticket. (The likelihood is 0 for n < m, 1/n for n ≥ m, and this is greatest when n = m. Note that the maximum likelihood estimate of n occurs at the lower extreme of possible values {m, m + 1, ...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The expected value of the number m on the drawn ticket, and therefore the expected value of ${\hat {n}}$ , is (n + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for n will systematically underestimate n by (n − 1)/2.

Discrete distribution, finite parameter space

Suppose one wishes to determine just how biased an unfair coin is. Call the probability of tossing a HEAD p. The goal then becomes to determine p.

Suppose the coin is tossed 80 times: i.e., the sample might be something like x₁ = H, x₂ = T, …, x₈₀ = T, and the count of the number of HEADS "H" is observed.

The probability of tossing TAILS is 1 − p (so here p is θ above). Suppose the outcome is 49 HEADS and 31 TAILS, and suppose the coin was taken from a box containing three coins: one which gives HEADS with probability p = 1/3, one which gives HEADS with probability p = 1/2 and another which gives HEADS with probability p = 2/3. The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation the coin that has the largest likelihood can be found, given the data that were observed. By using the probability mass function of the binomial distribution with sample size equal to 80, number successes equal to 49 but different values of p (the "probability of success"), the likelihood function (defined below) takes one of three values:

{\begin{aligned}\Pr(\mathrm {H} =49\mid p=1/3)&={\binom {80}{49}}(1/3)^{49}(1-1/3)^{31}\approx 0.000,\\[6pt]\Pr(\mathrm {H} =49\mid p=1/2)&={\binom {80}{49}}(1/2)^{49}(1-1/2)^{31}\approx 0.012,\\[6pt]\Pr(\mathrm {H} =49\mid p=2/3)&={\binom {80}{49}}(2/3)^{49}(1-2/3)^{31}\approx 0.054.\end{aligned}}

The likelihood is maximized when p = 2/3, and so this is the maximum likelihood estimate for p.

Discrete distribution, continuous parameter space

Now suppose that there was only one coin but its p could have been any value 0 ≤ p ≤ 1. The likelihood function to be maximised is

L(p)=f_{D}(\mathrm {H} =49\mid p)={\binom {80}{49}}p^{49}(1-p)^{31},

and the maximisation is over all possible values 0 ≤ p ≤ 1.

likelihood function for proportion value of a binomial process (n = 10)

One way to maximize this function is by differentiating with respect to p and setting to zero:

{\begin{aligned}{0}&{}={\frac {\partial }{\partial p}}\left({\binom {80}{49}}p^{49}(1-p)^{31}\right),\\[8pt]{0}&{}=49p^{48}(1-p)^{31}-31p^{49}(1-p)^{30}\\[8pt]&{}=p^{48}(1-p)^{30}\left[49(1-p)-31p\right]\\[8pt]&{}=p^{48}(1-p)^{30}\left[49-80p\right]\end{aligned}}

which has solutions p = 0, p = 1, and p = 49/80. The solution which maximizes the likelihood is clearly p = 49/80 (since p = 0 and p = 1 result in a likelihood of zero). Thus the maximum likelihood estimator for p is 49/80.

This result is easily generalized by substituting a letter such as t in the place of 49 to represent the observed number of 'successes' of our Bernoulli trials, and a letter such as n in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields the maximum likelihood estimator t / n for any sequence of n Bernoulli trials resulting in t 'successes'.

Continuous distribution, continuous parameter space

For the normal distribution ${\mathcal {N}}(\mu ,\sigma ^{2})$ which has probability density function

f(x\mid \mu ,\sigma ^{2})={\frac {1}{{\sqrt {2\pi \sigma ^{2}}}\ }}\exp {\left(-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}\right)},

the corresponding probability density function for a sample of n independent identically distributed normal random variables (the likelihood) is

f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma ^{2})=\prod _{i=1}^{n}f(x_{i}\mid \mu ,\sigma ^{2})=\left({\frac {1}{2\pi \sigma ^{2}}}\right)^{n/2}\exp \left(-{\frac {\sum _{i=1}^{n}(x_{i}-\mu )^{2}}{2\sigma ^{2}}}\right),

or more conveniently:

f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma ^{2})=\left({\frac {1}{2\pi \sigma ^{2}}}\right)^{n/2}\exp \left(-{\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}+n({\bar {x}}-\mu )^{2}}{2\sigma ^{2}}}\right),

where ${\bar {x}}$ is the sample mean.

This family of distributions has two parameters: θ = (μ, σ), so we maximize the likelihood, ${\mathcal {L}}(\mu ,\sigma )=f(x_{1},\ldots ,x_{n}\mid \mu ,\sigma )$ , over both parameters simultaneously, or if possible, individually.

Since the logarithm function itself is a continuous strictly increasing function over the range of the likelihood, the values which maximize the likelihood will also maximize its logarithm (The likelihood's logarithm is not strictly increasing). This log likelihood can be written as follows:

\log({\mathcal {L}}(\mu ,\sigma ))=(-n/2)\log(2\pi \sigma ^{2})-{\frac {1}{2\sigma ^{2}}}\sum _{i=1}^{n}(x_{i}-\mu )^{2}

(Note: the log-likelihood is closely related to information entropy and Fisher information.)

We now compute the derivatives of this log likelihood as follows.

{\begin{aligned}0&={\frac {\partial }{\partial \mu }}\log({\mathcal {L}}(\mu ,\sigma ))=0-{\frac {-2n({\bar {x}}-\mu )}{2\sigma ^{2}}}.\end{aligned}}

This is solved by

{\hat {\mu }}={\bar {x}}=\sum _{i=1}^{n}{\frac {x_{i}}{n}}.

This is indeed the maximum of the function since it is the only turning point in μ and the second derivative is strictly less than zero. Its expectation value is equal to the parameter μ of the given distribution,

E\left[{\widehat {\mu }}\right]=\mu ,\,

which means that the maximum likelihood estimator ${\widehat {\mu }}$ is unbiased.

Similarly we differentiate the log likelihood with respect to σ and equate to zero:

{\begin{aligned}0&={\frac {\partial }{\partial \sigma }}\log \left(\left({\frac {1}{2\pi \sigma ^{2}}}\right)^{n/2}\exp \left(-{\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}+n({\bar {x}}-\mu )^{2}}{2\sigma ^{2}}}\right)\right)\\[6pt]&={\frac {\partial }{\partial \sigma }}\left({\frac {n}{2}}\log \left({\frac {1}{2\pi \sigma ^{2}}}\right)-{\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}+n({\bar {x}}-\mu )^{2}}{2\sigma ^{2}}}\right)\\[6pt]&=-{\frac {n}{\sigma }}+{\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}+n({\bar {x}}-\mu )^{2}}{\sigma ^{3}}}\end{aligned}}

which is solved by

{\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-\mu )^{2}.

Inserting the estimate $\mu ={\widehat {\mu }}$ we obtain

{\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}={\frac {1}{n}}\sum _{i=1}^{n}x_{i}^{2}-{\frac {1}{n^{2}}}\sum _{i=1}^{n}\sum _{j=1}^{n}x_{i}x_{j}.

To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables (statistical error) $\delta _{i}\equiv \mu -x_{i}$ . Expressing the estimate in these variables yields

{\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}(\mu -\delta _{i})^{2}-{\frac {1}{n^{2}}}\sum _{i=1}^{n}\sum _{j=1}^{n}(\mu -\delta _{i})(\mu -\delta _{j}).

Simplifying the expression above, utilizing the facts that $E\left[\delta _{i}\right]=0$ and $E[\delta _{i}^{2}]=\sigma ^{2}$ , allows us to obtain

E\left[{\widehat {\sigma }}^{2}\right]={\frac {n-1}{n}}\sigma ^{2}.

This means that the estimator ${\widehat {\sigma }}$ is biased. However, ${\widehat {\sigma }}$ is consistent.

Formally we say that the maximum likelihood estimator for $\theta =(\mu ,\sigma ^{2})$ is:

{\widehat {\theta }}=\left({\widehat {\mu }},{\widehat {\sigma }}^{2}\right).

In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.

The normal log likelihood at its maximum takes a particularly simple form:

\log({\mathcal {L}}({\hat {\mu }},{\hat {\sigma }}))={\frac {-n}{2}}(\log(2\pi {\hat {\sigma }}^{2})+1)

This maximum log likelihood can be shown to be the same for more general least squares, even for non-linear least squares. This is often used in determining likelihood-based approximate confidence intervals and confidence regions, which are generally more accurate than those using the asymptotic normality discussed above.

Non-independent variables

It may be the case that variables are correlated, that is, not independent. Two random variables X and Y are independent only if their joint probability density function is the product of the individual probability density functions, i.e.

f(x,y)=f(x)f(y)\,

Suppose one constructs an order-n Gaussian vector out of random variables $(x_{1},\ldots ,x_{n})\,$ , where each variable has means given by $(\mu _{1},\ldots ,\mu _{n})\,$ . Furthermore, let the covariance matrix be denoted by $\Sigma$ .

The joint probability density function of these n random variables is then given by:

f(x_{1},\ldots ,x_{n})={\frac {1}{(2\pi )^{n/2}{\sqrt {{\text{det}}(\Sigma )}}}}\exp \left(-{\frac {1}{2}}\left[x_{1}-\mu _{1},\ldots ,x_{n}-\mu _{n}\right]\Sigma ^{-1}\left[x_{1}-\mu _{1},\ldots ,x_{n}-\mu _{n}\right]^{T}\right)

In the two variable case, the joint probability density function is given by:

f(x,y)={\frac {1}{2\pi \sigma _{x}\sigma _{y}{\sqrt {1-\rho ^{2}}}}}\exp \left[-{\frac {1}{2(1-\rho ^{2})}}\left({\frac {(x-\mu _{x})^{2}}{\sigma _{x}^{2}}}-{\frac {2\rho (x-\mu _{x})(y-\mu _{y})}{\sigma _{x}\sigma _{y}}}+{\frac {(y-\mu _{y})^{2}}{\sigma _{y}^{2}}}\right)\right]

In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section Principles, using this density.

Iterative procedures

Consider problems where both states $x_{i}$ and parameters such as $\sigma ^{2}$ require to be estimated. Iterative procedures such as Expectation-maximization algorithms may be used to solve joint state-parameter estimation problems.

For example, suppose that n samples of state estimates ${\hat {x}}_{i}$ together with a sample mean ${\bar {x}}$ have been calculated by either a minimum-variance Kalman filter or a minimum-variance smoother using a previous variance estimate ${\widehat {\sigma }}^{2}$ . Then the next variance iterate may be obtained from the maximum likelihood estimate calculation

{\widehat {\sigma }}^{2}={\frac {1}{n}}\sum _{i=1}^{n}({\hat {x}}_{i}-{\bar {x}})^{2}.

The convergence of MLEs within filtering and smoothing EM algorithms has been studied in the literature.^[13]^[14]^[15]

Applications

Maximum likelihood estimation is used for a wide range of statistical models, including:

linear models and generalized linear models;
exploratory and confirmatory factor analysis;
structural equation modeling;
many situations in the context of hypothesis testing and confidence intervals;
discrete choice models;
signal detection (filtering).

These uses arise across applications in widespread set of fields, including:

communication systems;
psychometrics;
econometrics;
time-delay of arrival (TDOA) in acoustic or electromagnetic detection;
data modeling in nuclear and particle physics;
magnetic resonance imaging;^[16]^[17]
computational phylogenetics;
origin/destination and path-choice modeling in transport networks;
geographical satellite-image classification;
power system state estimation.

References

1 2 Pfanzagl, Johann, with the assistance of R. Hamböker (1994). Parametric statistical theory. Walter de Gruyter, Berlin, DE. pp. 207–208. ISBN 3-11-013863-8.
↑ Edgeworth & September 1908 and Edgeworth & December 1908
↑ Wilks, S. S. (1938). The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses. Annals of Mathematical Statistics, 9: 60–62. doi:10.1214/aoms/1177732360.
↑ Owen, Art B. (2001). Empirical Likelihood. London: Chapman & Hall/Boca Raton, FL: CRC Press. ISBN 978-1584880714.
↑ Wilks, Samuel S. (1962) Mathematical Statistics. New York: John Wiley & Sons. ISBN 978-0471946502.
↑ Savage (1976), Pratt (1976), Stigler (1978, 1986, 1999), Hald (1998, 1999), and Aldrich (1997)
↑ vinaitheerthan,renganathan (2017-05-12). "Maximum Likelihood Estimation and Likelihood Ratio Test revisited". www.vinaitheerthan.com. 1 (1).
↑ Pfanzagl (1994, p. 206)
↑ Newey & McFadden (1994, Theorem 2.5.)
↑ Lehmann & Casella (1998)
↑ Newey & McFadden (1994, Theorem 3.3.)
↑ Cox & Snell (1968, formula (20))
↑ Einicke, G.A.; Malos, J.T.; Reid, D.C.; Hainsworth, D.W. (January 2009). "Riccati Equation and EM Algorithm Convergence for Inertial Navigation Alignment". IEEE Trans. Signal Processing. 57 (1): 370–375. doi:10.1109/TSP.2008.2007090.
↑ Einicke, G.A.; Falco, G.; Malos, J.T. (May 2010). "EM Algorithm State Matrix Estimation for Navigation". IEEE Signal Processing Letters. 17 (5): 437–440. doi:10.1109/LSP.2010.2043151.
↑ Einicke, G.A.; Falco, G.; Dunn, M.T.; Reid, D.C. (May 2012). "Iterative Smoother-Based Variance Estimation". IEEE Signal Processing Letters. 19 (5): 275–278. doi:10.1109/LSP.2012.2190278.
↑ Sijbers, Jan; den Dekker, A.J. (2004). "Maximum Likelihood estimation of signal amplitude and noise variance from MR data". Magnetic Resonance in Medicine. 51 (3): 586–594. PMID 15004801. doi:10.1002/mrm.10728.
↑ Sijbers, Jan; den Dekker, A.J.; Scheunders, P.; Van Dyck, D. (1998). "Maximum Likelihood estimation of Rician distribution parameters". IEEE Transactions on Medical Imaging. 17 (3): 357–361. PMID 9735899. doi:10.1109/42.712125.

External links

Hazewinkel, Michiel, ed. (2001) [1994], "Maximum-likelihood method", Encyclopedia of Mathematics, Springer Science+Business Media B.V. / Kluwer Academic Publishers, ISBN 978-1-55608-010-4
Lemke, D. (2016). Maximum likelihood estimation and EM fixed point ideals for binary tensors. (San Francisco State University. Masters Theses Collection – Degree in Mathematics.). San Francisco, CA: [San Francisco State University].

OCLC# 980058195 URL: http://hdl.handle.net/10211.3/173201

Maximum Likelihood Estimation Primer (an excellent tutorial)
Implementing MLE for your own likelihood function using R
A selection of likelihood functions in R
"Tutorial on maximum likelihood estimation". Journal of Mathematical Psychology. CiteSeerX 10.1.1.74.671 .

Statistics

Descriptive statistics

Continuous data

Center	Mean arithmetic geometric harmonic Median Mode
Dispersion	Variance Standard deviation Coefficient of variation Percentile Range Interquartile range
Shape	Moments Skewness Kurtosis L-moments

Count data

Index of dispersion

Summary tables

Dependence

Graphics

Data collection

Study design	Population Statistic Effect size Statistical power Sample size determination Missing data
Survey methodology	Sampling stratified cluster Standard error Opinion poll Questionnaire
Controlled experiments	Design control optimal Controlled trial Randomized Random assignment Replication Blocking Interaction Factorial experiment
Uncontrolled studies	Observational study Natural experiment Quasi-experiment

Statistical inference

Statistical theory

Frequentist inference

Point estimation	Estimating equations Maximum likelihood Method of moments M-estimator Minimum distance Unbiased estimators Mean-unbiased minimum-variance Rao–Blackwellization Lehmann–Scheffé theorem Median unbiased Plug-in
Interval estimation	Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling Bootstrap Jackknife
Testing hypotheses	1- & 2-tails Power Uniformly most powerful test Permutation test Randomization test Multiple comparisons
Parametric tests	Likelihood-ratio Wald Score

Specific tests

Z (normal) Student's t-test F
Goodness of fit	Chi-squared Kolmogorov–Smirnov Anderson–Darling Lilliefors Jarque–Bera Normality (Shapiro–Wilk) Likelihood-ratio test Model selection Cross validation AIC BIC
Rank statistics	Sign Sample median Signed rank (Wilcoxon) Hodges–Lehmann estimator Rank sum (Mann–Whitney) Nonparametric anova 1-way (Kruskal–Wallis) 2-way (Friedman) Ordered alternative (Jonckheere–Terpstra)

Bayesian inference

Correlation	Pearson product-moment Partial correlation Confounding variable Coefficient of determination
Regression analysis	Errors and residuals Regression model validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS)
Linear regression	Simple linear regression Ordinary least squares General linear model Bayesian regression
Non-standard predictors	Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity
Generalized linear model	Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions
Partition of variance	Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

Categorical

Multivariate

Time-series

General	Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality
Specific tests	Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey
Time domain	Autocorrelation (ACF) partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR)
Frequency domain	Spectral density estimation Fourier analysis Wavelet

Survival

Survival function	Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time
Hazard function	Nelson–Aalen estimator
Test	Log-rank test

Applications

Biostatistics	Bioinformatics Clinical trials / studies Epidemiology Medical statistics
Engineering statistics	Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification
Social statistics	Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population statistics Psychometrics
Spatial statistics	Cartography Environmental statistics Geographic information system Geostatistics Kriging

Category
Portal
Commons
WikiProject

This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.

Maximum likelihood estimation

History

Principles

Properties

Consistency

Asymptotic normality

Estimate on boundary

Data boundary parameter-dependent

Nuisance parameters

Increasing information

Sketch of proof

Functional invariance

Higher-order properties

Examples

Discrete uniform distribution

Discrete distribution, finite parameter space

Discrete distribution, continuous parameter space

Continuous distribution, continuous parameter space

Non-independent variables

Iterative procedures

Applications

See also

References

Further reading

External links