Mallows' Cp
From Wikipedia, the free encyclopedia
Mallows' Cp is mostly used as a stopping rule for various forms of stepwise regression. Collinearity in a regression model results from the common mistake of putting too many regressors into the model. Inevitably many of the hopefully independent input variables will have effects that are too correlated and cannot be separately estimated. When there are too many regressors, variables whose coefficients must be estimated, in a regression model it is said to be "over-fit." The worst case of this is when the number of parameters to be estimated is larger than the number of observations so that some effects cannot be estimated at all. Mallows proposed his Cp statistic as a subsetting criterion to be used in selecting a reduced model without such problems. If P regressors are selected from a set of K>P, Cp is defined as:
( SSEp / S2 ) - N + 2P, where
SSEp is the Sum[i=1,N] of (Yi-Ypi)2, the error sum of squares for the model with P regressors, Ypi being the ith predicted value of Y from the P regressors, S2 is the residual mean square after regression on the complete set of K regressors, and N is the sample size. If the model used to form S2 fits without bias, then S2Cp is an unbiased estimator of the mean squared prediction error (MSPE).
[edit] Practical use
A common misconception is that the "best" model is the minimizer of Cp. While true that for independent Gaussian errors of constant variance, the model minimizing MSPE is in some sense optimal, this is not necessarily true for Cp. Rather, because Cp is a random variable, it is important to consider its distribution. For example, one may form confidence intervals for Cp under its null distribution; that is, when the bias is zero. Cp is similar to the Akaike Information Criterion and, as a reliable measure of the "goodness of fit" for a model, tends to be less dependent than R-square on the number of effects in the model. Hence, Cp tends to find the best subset that includes only the important predictors of the dependent variable. Under a model not suffering from appreciable lack of fit (bias), Cp is has expectation nearly equal to P; otherwise the expectation is roughly P plus a positive bias term. Nevertheless, even though it has expectation greater than or equal to P, there is nothing to prevent Cp<P or even Cp<0 in extreme cases. Thus, a rule of thumb is to select a model in which the value of Cp is close to the number of terms, including any constant term, in the model. Frequently, the smallest subset model such that 2P>Cp>P with acceptible "p-values" for all model parameter estimates works best.
[edit] References
- Mallows, C.L. (1973) "Some Comments on Cp" Technometrics, 15, 661-675.
- Hocking, R.R. (1976) "The Analysis and Selection of Variables in Linear Regression" Biometrics, 32, 1-50.
- Daniel, C. and Wood, F. (1980) Fitting Equations to Data, Rev. Ed., NY: Wiley & Sons, Inc.
