Talk:Principle of maximum entropy

From Wikipedia, the free encyclopedia

	Physics Portal This article is within the scope of WikiProject Physics, which collaborates on articles related to physics.
???	This article has not yet received a rating on the assessment scale. [FAQ]
???	This article has not yet received an importance rating within physics.
Help with this template Please rate this article, and then leave comments here to explain the ratings and/or to identify the strengths and weaknesses of the article.

I find this discussion very doctrinaire and probably incomprehensible to most mathematicians for lack of context; maybe I'll do some more substantive editing later. Michael Hardy 17:46 Mar 30, 2003

Doctrinaire - a person inflexibly attached to a practice or theory without regard to its practicality. Online dictionary definition. Hey, I just tried to describe what it is - whether or not it's valid is an issue that requires its own subsection. Since most of what I've read on the subject of the validity of PME was written by its proponents, I have the information to give only one side of the story (from as N a POV as I can manage).Cyan 07:37 Apr 1, 2003 (UTC)

I don't claim to be a mathematician, and yet with a few terms of calculus and discrete math under my belt I find this presentation to be very accessible. I don't see how it can be made any more accessible without sacrificing content. I learned a few new things from this page (ie proving that ME solution is also ML solution) that I haven't come across when browsing papers on maxent.

I have some knowledge on how the algorithms that approximate maximum entropy solution work (the GIS and the IIS), if there's demand for it, perhaps I should post some info? yaroslavvb Jun 3, 2003 (PST)

Absolutely. But the PME page is already rather long. I suggest you create a new page (or pages) for these algorithms, and provide links to and from the PME page. Cyan 04:35 5 Jun 2003 (UTC)

<Mild chagrin> See also the second rule 25 on Wikipedia Anti-Rules. Cyan 21:53 Apr 3, 2003 (UTC)

What I meant by "doctrinaire" is that it imitates closely the language of Edwin Jaynes and may be incomprehensible to those unfamiliar with Jaynes' writings. One of these days I'll edit this article, but for the Time Being I have other obligations. Michael Hardy 01:36 Apr 4, 2003 (UTC)

I think the minus sign on the equations to find lambda values is wrong. I'll remove it. --163.117.155.37 18:01, 12 January 2007 (UTC)

[edit] Epistemic probability?

I've never seen that term used. It seems out of place in a mathematical context, and more appropriate to philosophy. I recommend changing it to the more standard term "Bayesian".

Who are you, that the fact that YOU have never seen it should be considered significant? I think it conveys the idea better than "Bayesian". Michael Hardy 21:12, 11 October 2005 (UTC)

I think it's a good term here, underlining that we're talking about probabilities being used to handle a lack of complete knowledge. Bayesian writers are often keen to stress that Bayesian inference and Bayesian methods are part of epistemology -- ie how to handle knowledge and incomplete knowledge; rather than ontology -- statements about the actual nature of the world. They are also clear on the value of always keeping the two clearly distinguished. Jheald 15:36, 20 October 2005 (UTC)

[edit] MLE and bayesianism

In the current article, we can read

maximum entropy principle is like other Bayesian methods in that

implying that MLE is a bayesian method. But, up to my knowledge, this claim is controversial (for instance R.Neal said :Maximum entropy is not consistent with Bayesian methods). Should we modify this sentence? For more information on this debate, here is starting a discussion, with good pointers.--Dangauthier 16:48, 16 February 2007 (UTC)

Looked at your blog quickly, didn't work through the example, but the result looks very fishy to me.

Given that the Principle of Maximum Entropy with respect to an invariant measure is essentially the same thing as Kullback's Minimum Discrimination Information (MDI) principle, you might like to look at Relative_entropy#Principle_of_minimum_discrimination_information.

That seems to me to show why MDI should replicate Bayes where Bayes is applicable.

Can you diagnose why your example is different ? Jheald 17:49, 16 February 2007 (UTC)

Even Bayesians disagree what "Bayesian" means. IE, is MAP Bayesian? Or must inference about models use model averaging? There's no consensus. There have been papers showing how MaxEnt can be massaged to look like a special case of Bayesian approach, and vica versa, the whole disagreement is mostly about semantics. --yaroslavvb

There's another discussion of the example here with a useful contribution from John Baez.

Basically, there are two different scenarios that need to be distinguished: Does the constraint only apply to our prior probability assessment, so that once we have formed that prior probability, we can forget the constraint and just apply Bayes' theorem using that prior distribution? Or must the constraint also apply to the posterior distribution - in which case it has to be included as a nuisance variable in its own right in the model, and explicitly conditionalised on in a Bayesian inference. Jheald 23:47, 3 March 2007 (UTC)