Maximum a posteriori estimation

Bayesian statistics
Part of a series on Statistics

Theory
Admissible decision rule Bayesian efficiency Bayesian probability Probability interpretations Bayes' theorem Bayes' rule Bayes factor Bayesian inference Bayesian network Prior Posterior Likelihood Conjugate prior Posterior predictive Hyperparameter Hyperprior Principle of indifference Principle of maximum entropy Empirical Bayes method Cromwell's rule Bernstein–von Mises theorem Bayesian information criterion Credible interval Maximum a posteriori estimation
Techniques
Bayesian linear regression Bayesian estimator Approximate Bayesian computation
Statistics portal

In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is a mode of the posterior distribution. The MAP can be used to obtain a point estimate of an unobserved quantity on the basis of empirical data. It is closely related to Fisher's method of maximum likelihood (ML), but employs an augmented optimization objective which incorporates a prior distribution over the quantity one wants to estimate. MAP estimation can therefore be seen as a regularization of ML estimation.

Description

Assume that we want to estimate an unobserved population parameter $\theta$ on the basis of observations $x$ . Let $f$ be the sampling distribution of $x$ , so that $f(x|\theta)$ is the probability of $x$ when the underlying population parameter is $\theta$ . Then the function:

\theta \mapsto f(x | \theta) \!

is known as the likelihood function and the estimate:

\hat{\theta}_{\mathrm{ML}}(x) = \underset{\theta}{\operatorname{arg\,max}} \ f(x | \theta) \!

is the maximum likelihood estimate of $\theta$ .

Now assume that a prior distribution $g$ over $\theta$ exists. This allows us to treat $\theta$ as a random variable as in Bayesian statistics. Then the posterior distribution of $\theta$ is as follows:

\theta \mapsto f(\theta | x) = \frac{f(x | \theta) \, g(\theta)}{\displaystyle\int_{\vartheta \in \Theta} f(x | \vartheta) \, g(\vartheta) \, d\vartheta} \!

where $g$ is density function of $\theta$ , $\Theta$ is the domain of $g$ . This is a straightforward application of Bayes' theorem.

The method of maximum a posteriori estimation then estimates $\theta$ as the mode of the posterior distribution of this random variable:

\hat{\theta}_{\mathrm{MAP}}(x) = \underset{\theta}{\operatorname{arg\,max}} \ \frac{f(x | \theta) \, g(\theta)} {\displaystyle\int_{\vartheta} f(x | \vartheta) \, g(\vartheta) \, d\vartheta} = \underset{\theta}{\operatorname{arg\,max}} \ f(x | \theta) \, g(\theta). \!

The denominator of the posterior distribution (so-called marginal likelihood) does not depend on $\theta$ and therefore plays no role in the optimization. Observe that the MAP estimate of $\theta$ coincides with the ML estimate when the prior $g$ is uniform (that is, a constant function). And when the loss function is of the form:

L(\theta, a) = \begin{cases} 0 & \mbox{, if } |a-\theta|<c \\ 1 & \mbox{, otherwise} \\ \end{cases} \!

as $c$ goes to 0, the sequence of Bayes estimators approaches the MAP estimator, provided that the distribution of $\theta$ is unimodal. But generally a MAP estimator is not a Bayes estimator unless $\theta$ is discrete.

Computation

MAP estimates can be computed in several ways:

Analytically, when the mode(s) of the posterior distribution can be given in closed form. This is the case when conjugate priors are used.
Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
Via a Monte Carlo method using simulated annealing

Criticism

While MAP estimation is a limit of Bayes estimators (under the 0-1 loss function), it is not very representative of Bayesian methods in general. This is because MAP estimates are point estimates, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with credible intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively - which are more representative of typical loss functions - and because the posterior distribution may not have a simple analytic form: in this case, the distribution can be simulated using Markov chain Monte Carlo techniques, while optimization to find its mode(s) may be difficult or impossible.

An example of a density of a bimodal distribution in which the highest mode is uncharacteristic of the majority of the distribution

In many types of models, such as mixture models, the posterior may be multi-modal. In such a case, the usual recommendation is that one should choose the highest mode: this is not always feasible (global optimization is a difficult problem), nor in some cases even possible (such as when identifiability issues arise). Furthermore, the highest mode may be uncharacteristic of the majority of the posterior.

Finally, unlike ML estimators, the MAP estimate is not invariant under reparameterization. Switching from one parameterization to another involves introducing a Jacobian that impacts on the location of the maximum.^[1]

As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using an MAP estimate, consider the case where there is a need to classify inputs $x$ as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification $h_1$ , $h_2$ and $h_3$ with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, $x$ , $h_1$ classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier $h_1$ , $x$ is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify $x$ as negative.

Example

Suppose that we are given a sequence $(x_1, \dots, x_n)$ of IID $N(\mu,\sigma_v^2 )$ random variables and an a priori distribution of $\mu$ is given by $N(\mu_0,\sigma_m^2 )$ . We wish to find the MAP estimate of $\mu$ .

The function to be maximized is then given by

f(\mu) f(x | \mu)=\pi(\mu) L(\mu) = \frac{1}{\sqrt{2 \pi} \sigma_m} \exp\left(-\frac{1}{2} \left(\frac{\mu-\mu_0}{\sigma_m}\right)^2\right) \prod_{j=1}^n \frac{1}{\sqrt{2 \pi} \sigma_v} \exp\left(-\frac{1}{2} \left(\frac{x_j - \mu}{\sigma_v}\right)^2\right),

which is equivalent to minimizing the following function of $\mu$ :

\sum_{j=1}^n \left(\frac{x_j - \mu}{\sigma_v}\right)^2 + \left(\frac{\mu-\mu_0}{\sigma_m}\right)^2.

Thus, we see that the MAP estimator for μ is given by

\hat{\mu}_{MAP} = \frac{n \sigma_m^2}{n \sigma_m^2 + \sigma_v^2 } \left(\frac{1}{n} \sum_{j=1}^n x_j \right) + \frac{\sigma_v^2}{n \sigma_m^2 + \sigma_v^2 } \mu_0.

which turns out to be a linear interpolation between the prior mean and the sample mean weighted by their respective covariances.

The case of $\sigma_m \to \infty$ is called a non-informative prior and leads to an ill-defined a priori probability distribution; in this case $\hat{\mu}_{MAP} \to \hat{\mu}_{ML}.$

References

↑ Murphy, Kevin P. (2012). Machine learning : a probabilistic perspective. Cambridge, MA: MIT Press. pp. 151–152. ISBN 0262018020.

M. DeGroot, Optimal Statistical Decisions, McGraw-Hill, (1970).
Harold W. Sorenson, (1980) "Parameter Estimation: Principles and Problems", Marcel Dekker.

Statistics

Descriptive statistics

Continuous data

Location	Mean arithmetic geometric harmonic Median Mode

Dispersion	Range Standard deviation Coefficient of variation Percentile Interquartile range

Shape	Variance Skewness Kurtosis Moments L-moments

Count data

Index of dispersion

Summary tables

Dependence

Statistical graphics

Data collection

Study design	Effect size Standard error Statistical power Sample size determination

Survey methodology	Sampling stratified cluster Opinion poll Questionnaire

Controlled experiments	Design control optimal Controlled trial Randomized Random assignment Replication Blocking Factorial experiment

Uncontrolled studies	Observational study Natural experiment Quasi-experiment

Statistical inference

Statistical theory

Frequentist inference

Confidence interval Testing hypotheses Power

Unbiased estimators	Mean unbiased minimum-variance Median unbiased

Biased estimators	Maximum likelihood Method of moments Minimum distance Density estimation

Parametric tests	Likelihood-ratio Wald Score

Specific tests

Z (normal) Student's t-test F Shapiro–Wilk Kolmogorov–Smirnov

Goodness of fit	Chi-squared G Sample source (Anderson–Darling) Sample normality (Shapiro–Wilk) Skewness / kurtosis normality (Jarque-Bera) Model comparison (Likelihood-ratio) Model quality (Akaike criterion)

Signed-rank	1-sample (Wilcoxon) 2-sample (Mann–Whitney U) 1-way anova (Kruskal–Wallis)

Bayesian inference

Correlation	Pearson product–moment Partial correlation Confounding variable Coefficient of determination

Regression analysis	Errors and residuals Regression model validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS)

Linear regression	Simple linear regression Ordinary least squares General linear model Bayesian regression

Non-standard predictors	Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity

Generalized linear model	Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance	Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

Categorical

Multivariate

Time-series

General	Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality

Specific tests	Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey

Time domain	Autocorrelation (ACF) partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR)

Frequency domain	Spectral density estimation Fourier analysis Wavelet

Survival

Survival function	Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time

Hazard function	Nelson–Aalen estimator

Test	Log-rank test

Applications

Biostatistics	Bioinformatics Clinical trials / studies Epidemiology Medical statistics

Engineering statistics	Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification

Social statistics	Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population Psychometrics

Spatial statistics	Cartography Environmental statistics Geographic information system Geostatistics Kriging

Category
Portal
Commons
WikiProject

This article is issued from Wikipedia - version of the Monday, January 04, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.