Talk:Information theory and measure theory

From Wikipedia, the free encyclopedia

Contents 1 Entropy as a Measure Theoretic measure 2 Possible misstatement 3 Kullback–Leibler divergence 4 Mis-statement of entropy.

[edit] Entropy as a Measure Theoretic measure

Where is this in Reza? I could not find it.

[edit] Possible misstatement

I am uncomfortable with the phrase 'we find that Shannon's "measure" of information content satisfies all the postulates and basic properties of a formal measure over sets.' This may not be quite correct, as it is a signed measure, as explained below in the article. How should it be better worded? --130.94.162.64 21:18, 20 June 2006 (UTC)

signed measure is still a measure, so if that's the only objection, it should be ok. on the other hand, the section title suggests entropy is a measure, that doesn't seem right. Mct mht 02:49, 21 June 2006 (UTC)

Not as defined in Measure (mathematics). There a "measure" is defined clearly as non-negative. The trouble is that that two rvs that are unconditionally independent can become conditionally dependent given a third rv. An example is given in the article. Maybe we should start calling it a signed measure right off the bat. A measure is normally assumed positive if not specified otherwise. --130.94.162.64 19:09, 21 June 2006 (UTC)

Measure (mathematics) does say signed measure is a measure, as it should. similarly, one can talk about complex measures or operator valued measures. but yeah, specify that it is a signed measure is a good idea. Mct mht 19:44, 21 June 2006 (UTC)

in the same vein, i think the article confuses measure in the sense of information theory with measure in the sense of real analysis in a few places. Mct mht 03:02, 21 June 2006 (UTC)

There are two different senses of "measure" in the article. One is the abstract measure over sets which forms the analogy with joint entropy, conditional entropy, and mutual information. The other is the measures over which one integrates in the various formulas of information theory. Where is the confusion? --130.94.162.64 04:16, 21 June 2006 (UTC)

some language in the section is not clear:

If we associate the existence of sets and with arbitrary discrete random variables X and Y, somehow representing the information borne by X and Y, respectively, such that: whenever X and Y are independent, and...

Associate sets to random variables how? are they the supports of the random variables? what's the σ-algebra? what's meant by two random variable being independent? Mct mht 03:14, 21 June 2006 (UTC)

Just pretend that those sets exist. They are not the supports of the random variables. The sigma-algebra is the algebra generated by the operations of countable set union and intersection on those sets. See statistical independence. --130.94.162.64 03:34, 21 June 2006 (UTC)

I mean unconditionally independent. --130.94.162.64 03:49, 21 June 2006 (UTC)

so given a family of random variables, one associates, somehow, a family of sets. the σ-algebra is the one generated by these family (in the same way the open sets generate the Borel σ-algebra), or does one assume that, somehow, the family is already a σ-algebra? also the section seems to imply the Shannon entropy is a measure on the said σ-algebra, is that correct?

Yes. --130.94.162.64 04:32, 21 June 2006 (UTC)

then there seems to be, at least, two ways measure theory is applied in this context. first, in the sense that entropy is a measure on some, undefined, sets corresponding to random variables. second, one can talk about random variables on a fixed measure space, and define information theoretic objectes in terms of the given measure. that a fair statement? Mct mht 04:14, 21 June 2006 (UTC)

The σ-algebra is the one generated by the family of sets. (They are not already a σ-algebra.) And I believe that is a fairly reasonable statement if I understand it right. --130.94.162.64 04:29, 21 June 2006 (UTC)

There's still a lot of explaining to do; that's why the article has the expert tag. --130.94.162.64 04:32, 21 June 2006 (UTC)

thanks for the responses. Mct mht 04:35, 21 June 2006 (UTC)

We really do need expert help, though. --130.94.162.64 05:30, 21 June 2006 (UTC)

[edit] Kullback–Leibler divergence

Also, the Kullback–Leibler divergence should be explained here in a proper measure-theoretic framework. --130.94.162.64 21:27, 20 June 2006 (UTC)

[edit] Mis-statement of entropy.

I am deeply suspicious of the defintions given here; I think they're wrong. Normally, given a collection of measureable sets {X_i}, the entropy is defined by

S = −	∑	μ(Xi)logμ(Xi)
	i

Its critical to have the logarithm in there, otherwise things like the partition function (statistical mechanics) and etc. just fail to work correctly. See, for example, information entropy.

Also, signed measures are bizarre, and you shoul avoid using them if you cannot explain why they are needed in the theory.

Also, I assumed that when this article said "H(X)", I assumed it meant "entropy", but on second reading, I see that it does not actually say what the symbols H(X) and I(X) are, or what they mean. Without defining the terms, the article is unreadable/subject to (mis-)interpretation, as perhaps I'm doing ??

Finally, I'm unhappy that this article blatently contradicts the article on random variable as to the defintion of what a random variable is. This article states that a random variable is a "set", and that is most certainly not how I understand random variables.

linas 23:54, 22 June 2006 (UTC)