User:Petergans/c/sandbox

From Wikipedia, the free encyclopedia

Errors-in-variables, also known as Total least squares or Rigorous least squares, is a least squares data modeling technique in which observational errors on both dependent and independent variables are taken into account. It can be applies to both linear and non-linear models.

Contents

[edit] Linear model

In the least squares method of data modelling, the objective function, S,

S=\mathbf{r^TWr}

is minimized. In linear least squares the model is defined as a linear combination of parameters,\boldsymbol\beta, so the residuals are given by

\mathbf{r=y-X\boldsymbol\beta}

There a m observations, y and n parameters, β, (m>n). X is a m\timesn matrix whose elements are either constants or functions of the independent variables, x. The weight matrix, W, is, ideally, the inverse of the variance-covariance matrix, \mathbf M_y of the observations, yThe independent variables are assumed to be error-free. The parameter estimates are found by setting the gradient equations to zero, which results in the normal equations

\mathbf{X^TWX\boldsymbol\beta=X^T Wy} or, alternatively, \mathbf{X^TWX\Delta \boldsymbol\beta=X^T W \Delta y}

Now, suppose that both x and y are subject to error, with variance-covariance matrices \mathbf M_x and \mathbf M_y respectively. In this case the objective function can be written as

S=\mathbf{r_x^TM_x^{-1}r_x+r_y^TM_y^{-1}r_y}

where \mathbf r_x\, and \mathbf r_y\, are the residuals in x and y respectively. Clearly these residuals cannot be independent of each other, but they must be constrained by some kind of relationship. Writing the model function as \mathbf{f(r_x,r_y,\boldsymbol\beta)}, the constraints are expressed by m condition equations.[1]

\mathbf{F=\Delta y -\frac{\partial f}{\partial r_x} r_x-\frac{\partial f}{\partial r_y} r_y -X\Delta\boldsymbol\beta=0}

Thus, the problem is to minimize the objective function subject to the m constraints. It is solved by the use of Lagrange multipliers. After some algebraic manipulations,[2] the result is obtained.

\mathbf{X^TM^{-1}X\Delta \boldsymbol\beta=X^T M^{-1} \Delta y} , or alternatively \mathbf{X^TM^{-1}X \boldsymbol\beta=X^T M^{-1} y}

Where M is the variance-covariance matrix relative to both independent and dependent variables.

\mathbf{M=K_xM_xK_x^T+K_yM_yK_y^T;\ K_x=-\frac{\partial f}{\partial r_x},\ K_y=-\frac{\partial f}{\partial r_Y}}

[edit] Example

In practice these equations are easy to use. When the data errors are uncorrelated, all matrices M and W are diagonal. Then, take the example of straight line fitting.

f(x_i,\beta)=\alpha + \beta x_i\!

It is easy to show that, in this case

M_{ii}=\sigma^2_{y,i}+\beta^2 \sigma^2_{x,i}

showing how the variance at the ith point is determined by the variances of both independent and dependent variables and by the model being used to fit the data. The expression may be generalized by noting that the parameter β is the slope of the line.

M_{ii}=\sigma^2_{y,i}+\left(\frac{dy}{dx}\right)^2_i \sigma^2_{x,i}

An expression of this type is used in fitting pH titration data where a small error on x translates to a large error on y when the slope is large.

[edit] Non-linear model

For non-linear systems similar reasoning shows that the normal equations for an iteration cycle can be written as

\mathbf{J^TM^{-1}J\Delta \boldsymbol\beta=J^T M^{-1} \Delta y}

[edit] Geometrical interpretation

When the independent variable is error-free a residual represents the "vertical" distance between between observed and calculated data points. In total least squares a residual represents the distance between a calculated data point and the data curve. In fact, if the errors on both variables are the same, the line represents the shortest distance between the calculated data point and the curve of observed data, that is, the residual vector is perpendicular to the tangent of the curve..

[edit] References

  1. ^ W.E. Deming, Statistical Adjustment of Data, Wiley, 1943
  2. ^ P. Gans, Data Fitting in the Chemical Sciences, Wiley, 1992
  • S. V. Huffel and P. Lemmerling, Total Least Squares and Errors-in-Variables Modeling: Analysis, Algorithms and Applications. Dordrecht, The Netherlands: Kluwer Academic Publishers, 2002.
  • S. Jo and S. W. Kim, "Consistent normalized least mean square filtering with noisy data matrix," IEEE Trans. Signal Processing, vol. 53, no. 6, pp. 2112-2123, Jun. 2005.
  • R. D. DeGroat and E. M. Dowling, "The data least squares problem and channel equalization," IEEE Trans. Signal Processing, vol. 41, no. 1, pp. 407–411, Jan. 1993.
  • T. Abatzoglou and J. Mendel, "Constrained total least squares," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP’87), Apr. 1987, vol. 12, pp. 1485–1488.
  • P. de Groen "An introduction to total least squares," in Nieuw Archief voor Wiskunde, Vierde serie, deel 14, 1996, pp. 237-253 arxiv.org.
v • d • e
Least squares and regression analysis
Least squares
Regression analysis
Statistics
Applications