Talk:Covariance matrix

From Wikipedia, the free encyclopedia

[edit] New problem...

This article starts with a big mess of slashes and sigmas and brackets... not sure if this is just my browser rendering something in a strange manner (though I'm using firefox, so I expect I'm not the only one seeing this) or quite what it is. I don't know how to fix it either; maybe somebody else does?

—The preceding unsigned comment was added by 129.169.10.56 (talk) 16:32, 5 December 2006 (UTC).

[edit] Explanatory Formula

This formula looks at first sight very complicated. Actually its derivation is quite simple (for simplicity we assume μ to be 0,(just replace everywhere X by X - μ if you want)):

fix a direction in n dimensions (unit vector), let's call it u
project your data onto this direction (you get a number for each of your data vectors, or taking all together a set of scalar samples); you perform just a scalar product i.e.: $S (i) = (X (i), u)$
compute ordinary variance for this new set of scalar numbers

We are almost finished. Of course for every unit vector you get (in general) different values, so you do not have just one number like in the scalar case but a whole bunch of numbers (a continuum) parametrised by the unit vectors in n dimensions (actually only the direction counts, u and -u give the same value) Now comes the big trick. We do not have to keep this infinity of numbers, as you can see below all the information is contained in the covariance matrix (wow!)

${\rm var} {((u,X))} = E((u,X)^2) = E((u,X)(u,X)) = E(u^\top XX^\top u)$

Now because u is a constant we have:

$E(u^\top XX^\top u) = u^\top E(XX^\top)u$

${\rm var} {((u,X)} = u^\top E(XX^\top)u = u^\top \Sigma u$

and we are done... (easy, isn't it :)

[edit] Comments moved here to Talk page

I have moved the comments above to this discussion page for several reasons. The assertion that this very simple formula looks "very complicated" seems exceedingly silly and definitely not NPOV. Then whoever wrote it refers to "its derivation". That makes no sense. It is a definition, not an identity or any sort of proposition. What proposition that author was trying to derive is never stated. The writing is a model of unclarity. Michael Hardy 22:52 Mar 12, 2003 (UTC)

[edit] I was attempting to explain covariance matrix

Okay I have written the stuff above and would like to reply...

1) It may be true tha for someone familiar with statistics the covariance matrix is immediatly understandable... nonetheless for someone outside of the statistics world (like me) it looks complicated...

2) by derivation I actually meant Motivation, or some hint how one can understand the meaning of covariance matrix and what it can be used for.

If you just throw the formula at people most of them do not understand its meaning and it is not clear at all why a formula like this makes any sense...

Of course you can let everyone find out by him/herself but this is just waste of time since the underlying concept is very simple...I did not try to prove any statement.

3) what is a clear writing is certainly discutable ;-) maybe it is not as water proof and correct as yours, but the idea is quite clear and this is the only thing that counts...

[edit] Idea was is not clear

Except that the idea is not clear without considerable interpretation, and even then I'm not entirely sure of what was meant. I think that what was intended could certainly be said more clearly with far fewer words. I'll make some attempt at this within a few days. Michael Hardy 20:48 Mar 13, 2003 (UTC)

[edit] Another explanation

Okay I try to explain the idea more clearly:

if you have some set of vector measueremnts you can consider it as a cloud of points in n-dimensions. If you want to find something interesting about your data set you can look a the data from different directions, or what is essentially the same perform a projection into 1,2, or 3 dimensions. But there are many projections possible, which one to take ? Life is not enough to try them all ;-) One criterion you can apply (not the best one but is better than nothing and it works sometimes...) is to look at directions for which the data have large variance (this makes sense if you want to find the most "energetic" components...) I tried to explain that the covariance matrix is a tool (at least can be interpreted in this way) to represent the data variances in all possible drections in a effective and compact way. If you once understand this it is immediatly clear why it is useful to look for the eigenvalues and eigenvectors of the covariance matrix.

I can not expect to see your version of this ! ;-)

[edit] Rephrased

I'll try my hand at the above:

If you have many measurements of similar things (such as GNP and airplane passengers for several years), such lists of information can be manipulated with linear algebra as vectors of numbers. If graphs are drawn of all useful combinations of these numbers, the result is many separate points on many graphs. Comparisons between the numerous graphs can be done if all the numbers are placed within one graph with as many dimensions as there are data items, thus creating a "cloud" of points in this space.

For example, GNP of one country for several years can be shown on a page as a graph of years by values. That has only two dimensions. Airline passenger numbers can be shown to the side of that graph in a three dimensional display. Adding another measurement, such as electricity consumed, requires a fourth dimension which is often hard to visualise.

Finding interesting information about such combinations can be difficult. One issue to examine is which part of that cloud of points has the greatest difference. The covariance matrix is the result of calculating distances between all the data points in all directions, and producing the direction through the cloud which has the greatest differences. Recording the distances between different types of data within the cloud produces the covariance matrix, where larger numbers suggest greater relationships.

(should eigenvalues and eigenvectors be introduced here, or are those considered to be parts of calculations other than covariance?) SEWilco 18:55, 15 Jan 2004 (UTC)

[edit] Article adjusted

I have inserted into the article some language that I think addresses your point, which I still think was quite unclear as you wrote it originally. Michael Hardy 19:53 Mar 14, 2003 (UTC)

PS: "One criterion is ....", but some other criteria may exist. (My point here is that in standard English, "criterion" is the singular. Michael Hardy 19:54 Mar 14, 2003 (UTC)

[edit] Still not clear and lost illustration

Unfortunately I'm not very happy with your version of my explanation.

Such sort of explanations can be found in any book, and I never found them very helpful. You have completely lost the geometric picture which makes all clear and simple. To say that the matrix entries are covariances between the variables

does not explain anything, and actually makes it more complicated because as you said this depends on the basis...

I would prefer to let decide the people who visit this site by themselves

which version they find more illuminating... At the moment it is hardly possible because my version is quite hidden :-(

Ps.

With "criterion" you are right, I'm not very good in english...

(nonostante capisci quasi tutto, che miracolo !)

[edit] Still looks unclear

I can write a more leisurely explanation when I have time, but your version still looks unclear to me as it stands. Michael Hardy 22:43 Mar 17, 2003 (UTC)

[edit] Please explain what is defined

I think that it is better to explain the things one defines. The above explanations helped me more to understand the topic than the mathematical absolutely correct formulas one finds, when looking for "covariance matrix". To my opinion most of the people that use Wikipedia are interested in both versions, so they should see them at the same site and not hidden in the discussion group.

[edit] What is unclear?

Maybe you could be more specific on "looks unclear to me"... I tried to write as clearly as I only can, because I want that everyone understands it with ease. If there is something unclear to you maybe I could explain it better but at the moment I do not know what is unclear ?

[edit] Please label your comments

People, label yourself in your comments so we know who is talking. Also be a little more specific about what you are pointing at. There are too many "that", "I", and "you" for it to be clear who is talking about what. I suggest the four-tilde signature so the date is included. SEWilco 17:11, 15 Jan 2004 (UTC)

[edit] Yet Another Rephrasing

(SEWilco 08:39, 7 Jul 2004 (UTC)) Maybe something like this will be useful:

A vector is a list of numbers. The variance is the square of the difference between a number and an expected value, such as the variance of two lengths is an area the size of the square of the difference. The covariance matrix has a list of numbers along one side, and the other side has a list of the expected values for each listed value. Each position of the matrix is filled in with the square of the difference between those two numbers. The covariance matrix then contains the variance between each listed number and the expected values of all numbers in the list. This shows how different all numbers are from all the expected values.

[edit] Nonstandard notation?

I've never encountered the usage of $var(\textbf{X})$ for denoting the covariance matix. I've always used $\textbf{C}_X$ for this (and $\textbf{R}_X$ for autocorrelation matrix). This is standard notation pracise in the field of signal processing. --Fredrik Orderud 12:26, 20 Apr 2005 (UTC)

How about $\operatorname{cov}(X)$ ? Cburnett 14:24, Apr 20, 2005 (UTC)

I've now changed the notation to "cov", which is in accordance with mathworld. I've also added a separate "signal processing" section containing the different notation used there. --Fredrik Orderud 21:02, 28 Apr 2005 (UTC)

Pardon me, but who wrote the comments immediately above??? Michael Hardy 20:07, 28 Apr 2005 (UTC)

Sorry. It as me, and I forgot to sign. --Fredrik Orderud 21:02, 28 Apr 2005 (UTC)

Standard notation:

$\operatorname{var}(\textbf{X}) = E[(\textbf{X} - E[\textbf{X}])(\textbf{X} - E[\textbf{X}])^{T}]$

ALSO standard notation:

$\operatorname{cov}(\textbf{X}) = E[(\textbf{X} - E[\textbf{X}])(\textbf{X} - E[\textbf{X}])^{T}]$

ALSO standard notation:

$\operatorname{cov}(\textbf{X},\textbf{Y}) = E[(\textbf{X} - E[\textbf{X}])(\textbf{Y} - E[\textbf{Y}])^{T}]$ (the "cross-covariance" between two random vectors)

Unfortunately the first two of these usages jar with each other. The first and third are in perfect harmony. The first notation is found in William Feller's celebrated two-volume book on probability, which everybody is familiar with, so it's surprising that some people are suggesting it first appeared on Wikipedia. It's also found in some statistics texts, e.g., Sanford Weissberg's linear regression text. Michael Hardy 18:05, 28 Apr 2005 (UTC)

[edit] Horrible mess!!

This article was starting to become an examplar of crackpothood. Someone who apparently didn't like the opening paragraphs, instead of replacing them with other material, simply put the other material above those opening paragraphs, so that the article started over again, saying, in a later paragraph, "In statistics, a covariance matrix is ..." etc., and giving the same elaborate definition again, with stylistic differences. And that eventually became the second of FOUR such iterations, with stylistic differences! Other things were wrong too. Why, for example was there a "stub" notice?? There should have been a "cleanup" notice right at the top, instead of a "stub" notice at the bottom. Inline TeX often gets misaligned or appears far too big, or both, on many browsers, but it looks as if someone went through and put perfectly good-looking non-TeX inline mathematical notation with TeX (e.g., "an n × n matrix" ---> "an $n \times n$ matrix"). (Tex generally looks very good when "displayed", however. And when TeX used in the normal way, as opposed to its use on Wikipedia, there's certainly no problem with inline math notation.) Using lower-case letters for random variables is jarring, since in many cases one wants to write such things as

F_X(x) = Pr(X ≤ x)

and it is crucial to be careful about which of the "x"s above are capital and which are lower-case. The cleanup isn't finished yet .... Michael Hardy 19:16, 28 Apr 2005 (UTC)

[edit] Properties

I added the list of properties in the article. There was only two of them stated, and these properties should definitivley be on an article about cov and var matrices. --Steffen Grønneberg 14:06, 5 October 2005 (UTC)

Somehow I find it difficult to understand the 5th property:

5. $\operatorname{cov}(\mathbf{X},\mathbf{X}) = \operatorname{cov}(\mathbf{X},\mathbf{Y}) = \operatorname{cov}(\mathbf{Y},\mathbf{X})^\top$

Shouldn't the correct formula be:

5. $\operatorname{cov}(\mathbf{X},\mathbf{Y}) = \operatorname{cov}(\mathbf{Y},\mathbf{X})^\top$

, since no relationship between "X" and "Y" is defined? Does anyone have a reference on this? --Fredrik Orderud 12:01, 11 October 2005 (UTC)

Yep, my bad. Fixed it now. The reference I used is Multivariate Analysis by K. V. Mardia, J. T. Kent, J. M. Bibby. Its in chap. 3. (http://www.amazon.com/gp/product/0124712525/103-2355319-3731041?v=glance&n=283155&n=507846&s=books&v=glance) Is there a place where it's usual to cite this? --Steffen Grønneberg 23:10, 12 October 2005 (UTC)

Before I go make a fool of myself, shouldn't A and B be q x p matrices instead of p x q matrices? --The imp 15:32, 15 March 2006 (UTC)

[edit] Conflict of var and cov?

I don't see how var(X,Y) and cov(X,Y) "conflict". We don't say that gamma(x) and x! or asin and arcsin "conflict" or "jar". It is perfectly OK to have var(X)=cov(X)=cov(X,X). The fact that there is a one-argument function cov doesn't exclude there being a two-argument, related, function. Consider, say, Γ(x)=Γ(x,0). --Macrakis 21:51, 21 December 2005 (UTC)

[edit] Calculation of covariance matrix

It would be nice to have a section on computational methods for calculating covariance matrices. I am unfortunately not competant to write it.... Any volunteers? --Macrakis 21:51, 21 December 2005 (UTC)

[edit] Possible covariance matrices

What are the restrictions on what matrices can be covariance matrices? I guess the matrix has to be symmetric; is any symmetric matrix a possible covariance matrix? --Trovatore 23:11, 19 June 2006 (UTC)

A square matrix with real entries is a covariance matrix if and only if it is non-negative definite. If X is a column vector-valued random variable, then the expected value of XX^T is the covariance matrix of the scalar compoments of X, so it should be clear why that has to be non-negative definite. By the spectral theorem in its finite-dimensional version, every non-negative definite real matrix M has a non-negative definite real square root, which let us call M^1/2. Then let X be any column vector of the right size whose entries are random variables the variance of each of which is 1 and the covariance between any two of which is 0. Then the covariance matrix of the entries in X is M. So any non-negative definite matrix is a covariance matrix. Michael Hardy 17:09, 20 June 2006 (UTC)

Suppose you know all diagonal entries and some off-diagonal, and you want to generate a nonnegative-definite symmetric matrix having those entries. Is there an efficient way to generate such a thing, in the large-finite-dimensional case? --Trovatore 22:48, 23 June 2006 (UTC)

Retrieved from "http://en.wikipedia.org../../../c/o/v/Talk%7ECovariance_matrix_eab0.html"