Residual sum of squares

In statistics, the residual sum of squares (RSS) is the sum of squares of residuals. It is also known as the sum of squared residuals (SSR) or the sum of squared errors of prediction (SSE). It is a measure of the discrepancy between the data and an estimation model. A small RSS indicates a tight fit of the model to the data.

In general, total sum of squares = explained sum of squares + residual sum of squares. For a proof of this in the multivariate ordinary least squares (OLS) case, see partitioning in the general OLS model.

One explanatory variable

In a model with a single explanatory variable, RSS is given by

RSS = \sum_{i=1}^n (y_i - f(x_i))^2,

where yi is the i th value of the variable to be predicted, xi is the i th value of the explanatory variable, and f(x_i) is the predicted value of yi (also termed \hat{y_i}). In a standard linear simple regression model, y_i = a+bx_i+\varepsilon_i\,, where a and b are coefficients, y and x are the regressand and the regressor, respectively, and ε is the error term. The sum of squares of residuals is the sum of squares of estimates of εi; that is

RSS = \sum_{i=1}^n (\varepsilon_i)^2 = \sum_{i=1}^n (y_i - (\alpha + \beta x_i))^2,

where \alpha is the estimated value of the constant term a and \beta is the estimated value of the slope coefficient b.

Matrix expression for the OLS residual sum of squares

The general regression model with n observations and k explanators, the first of which is a constant unit vector whose coefficient is the regression intercept, is

y = X \beta + e

where y is an n × 1 vector of dependent variable observations, each column of the n × k matrix X is a vector of observations on one of the k explanators, \beta is a k × 1 vector of true coefficients, and e is an n× 1 vector of the true underlying errors. The ordinary least squares estimator for \beta is

\hat \beta = (X^T X)^{-1}X^T y.

The residual vector \hat e is y - X \hat \beta = y - X (X^T X)^{-1}X^T y, so the residual sum of squares \hat e ^T \hat e is, after simplification,

RSS = y^T y - y^T X(X^T X)^{-1} X^T y = y^T [I - X(X^T X)^{-1} X^T] y = y^T [I - H] y , where H is the hat matrix, or the prediction matrix in linear regression.

See also

References