Deviance (statistics)

From Wikipedia, the free encyclopedia

Not to be confused with Deviation (statistics).

In statistics, deviance is a quality of fit statistic for a model that is often used for statistical hypothesis testing. It is a generalization of the idea of using the sum of squared residuals in ordinary least squares to cases where model-fitting is achieved by maximum likelihood.

Definition

The deviance for a model M₀, based on a dataset y, is defined as:^[1]^[2]

$D(y)=-2{\Big (}\log {\big (}p(y|{\hat \theta }_{0}){\big )}-\log {\big (}p(y|{\hat \theta }_{s}){\big )}{\Big )}.\,$

Here ${\hat \theta }_{0}$ denotes the fitted values of the parameters in the model M₀, while ${\hat \theta }_{s}$ denotes the fitted parameters for the "full model" (or "saturated model"): both sets of fitted values are implicitly functions of the observations y. Here the full model is a model with a parameter for every observation so that the data are fitted exactly. This expression is simply −2 times the log-likelihood ratio of the reduced model compared to the full model. The deviance is used to compare two models – in particular in the case of generalized linear models where it has a similar role to residual variance from ANOVA in linear models (RSS).

Suppose in the framework of the GLM, we have two nested models, M₁ and M₂. In particular, suppose that M₁ contains the parameters in M₂, and k additional parameters. Then, under the null hypothesis that M₂ is the true model, the difference between the deviances for the two models follows an approximate chi-squared distribution with k-degrees of freedom.^[2]

Some usage of the term "deviance" can be confusing. According to Collett:^[3]

"the quantity $-2\log {\big (}p(y|{\hat \theta }_{0}){\big )}$ is sometimes referred to as a deviance. This is [...] inappropriate, since unlike the deviance used in the context of generalized linear modelling, $-2\log {\big (}p(y|{\hat \theta }_{0}){\big )}$ does not measure deviation from a model that is a perfect fit to the data." However, since the principal use is in the form of the difference of the deviances of two models, this confusion in definition is unimportant.

Notes

↑ Nelder, J.A.; Wedderburn, R.W.M. (1972). "Generalized Linear Models". Journal of the Royal Statistical Society. Series A (General) 135 (3): 370–384. doi:10.2307/2344614. JSTOR 2344614.
↑ 2.0 2.1 McCullagh and Nelder (1989)
↑ Collett (2003)

References

McCullagh, Peter; Nelder, John (1989). Generalized Linear Models, Second Edition. Chapman & Hall/CRC. ISBN 0-412-31760-5.

Collett, David (2003). Modelling Survival Data in Medical Research, Second Edition. Chapman & Hall/CRC. ISBN 1-58488-325-1.

External links

Generalized Linear Models - Edward F. Connor
Lectures notes on Deviance

Statistics

Descriptive statistics

Continuous data

Location	Mean (Arithmetic, Geometric, Harmonic) Median Mode

Dispersion	Range Standard deviation Coefficient of variation Percentile Interquartile range

Shape	Variance Skewness Kurtosis Moments L-moments

Count data

Index of dispersion

Summary tables

Dependence

Statistical graphics

Data collection

Designing studies	Effect size Standard error Statistical power Sample size determination

Survey methodology	Sampling Stratified sampling Cluster sampling Opinion poll Questionnaire

Controlled experiment	Design of experiments Randomized experiment Random assignment Replication Blocking Factorial experiment Optimal design

Uncontrolled studies	Natural experiment Quasi-experiment Observational study

Statistical inference

Statistical theory	Sampling distribution Order statistic Scan statistic Record value Sufficiency Completeness Exponential family Permutation test (Randomization test) Empirical distribution Bootstrap U statistic Efficiency Asymptotics Robustness

Frequentist inference	Unbiased estimator (Mean unbiased minimum variance, Median unbiased) Biased estimators (Maximum likelihood, Method of moments, Minimum distance, Density estimation) Confidence interval Testing hypotheses Power Parametric tests (Likelihood-ratio, Wald, Score)

Specific tests	Z (normal) Student's t-test F Goodness of fit (Chi-squared, G, Sample source, sample normality, Skewness & kurtosis Normality, Model comparison, Model quality) Signed-rank (1-sample, 2-sample, 1-way anova) Shapiro–Wilk Kolmogorov–Smirnov

Bayesian inference	Bayesian probability Prior Posterior Credible interval Bayes factor Bayesian estimator Maximum posterior estimator

Correlation and regression analysis

Correlation	Pearson product–moment correlation Partial correlation Confounding variable Coefficient of determination

Regression analysis	Errors and residuals Regression model validation Mixed effects models Simultaneous equations models MARS

Linear regression	Simple linear regression Ordinary least squares General linear model Bayesian regression

Non-standard predictors	Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity

Generalized linear model	Exponential families Logistic (Bernoulli) Binomial Poisson

Partition of variance	Analysis of variance (ANOVA) Analysis of covariance Multivariate ANOVA Degrees of freedom

Categorical, multivariate, time-series, or survival analysis

Categorical data

Multivariate statistics

Time series analysis

General	Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration

Specific tests	Granger causality Q-Statistic Durbin–Watson

Time domain	ACF PACF XCF ARMA model ARIMA model ARCH Vector autoregression

Frequency domain	Spectral density estimation Fourier analysis

Survival analysis

Applications

Biostatistics	Bioinformatics Clinical trials & studies Epidemiology Medical statistics

Engineering statistics	Chemometrics Methods engineering Probabilistic design Process & Quality control Reliability System identification

Social statistics	Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population Psychometrics

Spatial statistics	Cartography Environmental statistics Geographic information system Geostatistics Kriging

Category
Portal
Outline
Index

Least squares and regression analysis

Computational statistics

Correlation and dependence

Regression analysis

Ordinary least squares
Partial least squares
Total least squares
Ridge regression

Regression as a
statistical model

Linear regression	Simple linear regression Ordinary least squares Generalized least squares Weighted least squares General linear model

Predictor structure	Polynomial regression Growth curve Segmented regression Local regression

Non-standard	Nonlinear regression Nonparametric Semiparametric Robust Quantile Isotonic

Non-normal errors	Generalized linear model Binomial Poisson Logistic

Decomposition of variance

Model exploration

Background

Design of experiments

Numerical approximation