Talk:Linear least squares

From Wikipedia, the free encyclopedia

Contents 1 Relevancy 2 Normal Equation and Normal Matrix 3 Help! We're being outgunned and overrun by helpful mathematicians... 4 Overexplaining?

[edit] Relevancy

03:33, 26 October 2005 Oleg Alexandrov m (I don't find the octave code particularly relevant;)

It's very relevant. Actually calculating this is interesting, not just knowing how to calculate it with pen and paper. --marvinXP (talk)

You are reffering to this.

I am myself doing numerical analysis for living, use Matlab (of whom Octave is a clone) and have no bias against numerical programs. However, that piece of code is just rewiting the formula

from the article in Octave's notation. It is a trivial exercise I would say, and not worth its place in the article. Oleg Alexandrov (talk) 06:04, 26 October 2005 (UTC)

I agree. Actually, I would have deleted the Octave fragment if I had time to do so before Oleg did. -- Jitse Niesen (talk) 11:10, 26 October 2005 (UTC)

I see your point, but I disaree. A non-mathematician may want to actually use the formula. It's a very practical application. It's not a matter of "this is how it's written in program X", but rather "if you have the values, you can plug it into this free program, and have the result presented to you". I'd say most people who'd want to calculate this (let's assume they know of least squares), may not know octave, or matlab or any other math-program besides a simple calculator. And may not afford mathlab. --marvinXP (talk), 16:24, 29 October 2005 (UTC)

Even if it is true that most people who know about least squares and want to calculate a least-squares solution, do not know that there are programs for doing so (which I doubt), it would be irrelevant for this article, because it's not a statement specifically about least squares. Many mathematical concepts can be calculated by programs, free or otherwise. I think it would be silly to add this fact to articles like matrix multiplication and matrix inversion. By the way, your Octave fragment is suboptimal. As a general rule, one should try to avoid inverting a matrix. The proper way is to use the slash, as follows:

A = [0,1; 2,1; 4,1; -1,1]

b = [3; 3; 4; 2]

x = A \ b

-- Jitse Niesen (talk) 17:51, 29 October 2005 (UTC)

I'd consider matrix multiplication less directly applicable, and would just involve defining '*' in a given program. This example had more steps to it (well it had before you showed the backslash operator, thank btw). But I guess I'll concede. --marvinXP (talk), 23:09, 29 October 2005 (UTC)

[edit] Normal Equation and Normal Matrix

The normal equation forms the matrix product of A-transpose and A. This forms a new, square matrix C. This new square matrix (C) is a Normal matrix. A Normal matrix has the property that its product with its transpose is the same whether pre or post multiplied. This makes/ensures the matrix symetric and at least positive semidefinate (usually positive definate).

It is easy to confuse the form for the normal equation and normal matrix if both refer to a generic matrix using the same symbol 'A'. It is the symetric positive semidefinate property (and its consequences) that is 'Normal'.

Philip Oakley

[edit] Help! We're being outgunned and overrun by helpful mathematicians...

In that this is an encyclopedia, i.e. a place where people go to understand concepts they currently don't, I feel that the "explanation" of this term is utterly too complex and filled with mathematical jargon. Not that it doesn't belong, but there needs to be introductory material geared towards newcomers to this material. If I wanted a mathematical proof or advanced applications I would probably consult a book on statistics.

As an example of a better and more appropriate introduction geared towards mathematical newcomers, the wikipedia entry for "Method of least squares" strikes me as well written and clear. —The preceding unsigned comment was added by 161.55.228.176 (talk) 18:17, 28 September 2006 (UTC).

Clarity should definitely come before technicalities, but I think linear least squares is a specific example of an application of the method of least squares, so it is only natural that the former should have more technical detail. The lead paragraph, though, could use some more background, as well as a link to method of least squares. --Zvika 17:02, 28 September 2006 (UTC)

I couldn't agree more with the immediately foregoing comments. One thing that would help greatly IMHO would be to use standard notation, consistent with the usual applications of these ideas, as shown in Regression analysis. It is usual to refer to the data or inputs as the X matrix, the unknown parameters as Beta or B, and the dependent or outputs as Y. Accordingly, the normal equations should be expressed in a format like:

                             -.
 Y  =  X B  <=>  B  =  (X' X)   X' Y
n 1   n k 1     k 1   k   n  k    n 1

where the subscripts are the row and column dimensions, identified to show how they must match, and -. indicates generalized inverse. --Mbhiii 18:48, 27 October 2006 (UTC)

I have seen both the syntax and the syntax used in books. As long as the article is consistent, I don't think it necessarily requires one syntax over another. This subscript notation that you suggest is new to me, though. Is this something you've seen before? I am afraid it might be more confusing than helpful if the reader is not familiar with the notation. --Zvika 21:41, 28 October 2006 (UTC)

Internal consistency should not be the only criterion, but external consistency, as much as possible, with the dominant practical uses of the day, should be very important, as well, because it increases recognizability and, therefore, the general utility of the article. The intersubjective standard of reference that Wikipedia represents, and is slowly increasing, requires no less. This argues for the X matrix form. I know the subscript notation is new, but it's very useful. It's a revision of Einstein's matrix notation. (He used one subscript and one superscript per matrix, which conflicted with exponents.) Though very gifted in abstract visualization, he was not always the best student, and noncommutative matrix multiplication gave him headaches in long calculations. This cleaned up version of his notation is gaining acceptance among those teaching matrix algebra and its applications. Simply identify the left subscript as the number of rows and the right one as number of columns, and all is clear. --Mbhiii 14:28, 31 October 2006 (UTC)

I don't understand how you can say we need external consistency, and then push for a new notation. By the way, Einstein's index notation does not show the dimensions of the matrices, so it's something completely different. -- Jitse Niesen (talk) 00:49, 1 November 2006 (UTC)

An X matrix form for the normal equations increases the article's external consistency. Revised Einstein matrix notation is for clarity, an aid to people like students or programmers who don't necessarily use matrix algebra all the time. By the way, Einstein matrix notation is not Einstein summation notation. --Mbhiii 13:55, 1 November 2006 (UTC)

[edit] Overexplaining?

Zvika, perhaps I am overexplaining, but as a master's in physics I didn't see the equality of the middle terms right away. It seemed useful - to me, and probably to some other users as well - to explain that step it in some more detail. Pallas44 13:06, 16 November 2006 (UTC)

Sorry... it still seems pretty obvious to me. It's just a result of the fact that for any two vectors, a^Tb = b^Ta. Right in the next sentence we differentiate a vector equation, equate to zero and give the result - a far more complicated procedure which is given without any technical details. These are things the reader is assumed to be capable of doing by himself or herself, if necessary, or to just take our word for it if he/she is in a hurry. However, I will be happy to hear other people's opinions. We are talking about this edit. --Zvika 20:09, 20 November 2006 (UTC)