Talk:Bayesian probability

From Wikipedia, the free encyclopedia

This article is within the scope of WikiProject Statistics, which collaborates to improve Wikipedia's coverage of statistics. If you would like to participate, please visit the project page.
WikiProject Mathematics
Mathematics Portal
This article is within the scope of WikiProject Mathematics, which collaborates on articles related to mathematics.
Mathematics rating: Start Class High Priority  Field: Probability and statistics
One of the 500 most frequently viewed mathematics articles.
Archive
Archives
Archive 1 Archive 2


Contents

[edit] Too technical?

I removed the to technical tag from this page. It is the opposite of to technical. It almost needs a more mathematically technical treatment in atleast one section. —The preceding unsigned comment was added by Jeremiahrounds (talk • contribs) 09:36, 22 June 2007.

[edit] What is this article about???

Can anyone clarify what this article is about? Logicus writes <quote>this article is NOT about 'Bayesian STATISTICS', but rather about 'Bayesian PROBABILITY'</quote>, and removes a reference to Cox's axioms. In what sense, may I ask, is Bayesian probability unrelated to Bayesian statistics? This article is a complete mess!!!Tomixdf (talk) 21:40, 4 March 2008 (UTC)

I agree. Cox's axioms (and theorems) are obviously related to Bayesian PROBABILITY, since they claim to provide an axiomatic basis for it. Bill Jefferys (talk) 00:44, 5 March 2008 (UTC)
At one point Logicus mentions "the article's definition of Bayesian probability as subjective epistemic probabilism". It seems the article should be renamed to reflect what it is actually about then (ie. subjective epistemic probabilism, or so it seems). That would be reasonable since there is a fairly decent article on "Bayesian inference", and this article seems to be redundant. In that case, I'll direct my efforts to adding a history section there, instead of trying to improve this mess. Tomixdf (talk) 06:26, 5 March 2008 (UTC)


Logicus comments: I agree with Tomixdf that this article is currently a terrible conceptually confused mess. I tried to sort this confusion out some time ago in creating the Talk Section 21 ‘What is probability ?’ last October. But rather than its resulting in historically informed logical clarifications of the different conceptions of the term ‘probability’ in order to sort out this confusion, rather it promoted what I regarded as the authoritarian mystical ravings of a radical Jaynesian who apparently did not even understand Jaynes was an objectivist probabilist whose views were therefore irrelevant to an article on subjective probability as the article then (mistakenly) defined ‘Bayesian probability’, other than to mention a contrasting alternative philosophy of probability in passing. So I gave up.
To correct yours and Jefferys’ misinterpretation of why I removed the statement about Bayesian statistics in the introduction, it was not because I think Bayesian probability is unrelated to Bayesian statistics as you surmise, but rather because if the article is to be about subjective probability as it was defined to be, then Bayesian statistics that are the same in both objective and subjective probability and indeed even in non-epistemic material probability a la Kolmogorov etc are irrelevant to the issue of explicating the specific subjective epistemic notion of probability as ‘strength of belief in the truth of a proposition’, as distinct from such as objectivist epistemic ‘degree of certainty of a proposition’, for example.
By the way, Jefferys is on record on these pages as rejecting epistemic probability in his probabilist philosophy of science in favour of a subjectivist realist fallibilist instrumentalist concept of probability which maintains all scientific laws are false but may be more or less useful instruments of prediction, and that ‘probability’ is just the degree to which they are believed to be useful for making predictions, rather than the degree of belief that they are true. Certainly realist instrumentalist fallibilism seems to be the dominant philosophy of science amongst scientists, and Jefferys claims to be an astronomer. Realist fallibilist instrumentalism is to be distinguished from idealist instrumentalism which holds that scientific theories are neither nor false (idealism) but are useful logical classifying devices and instruments of prediction, a position that such as the physicist Duhem equivocated on. Scientific realism holds that scientific theories do have truth values, being either true or false contrary to idealism's claim that they are neither, and realist fallibilism holds that in particular this value is always the value 'false'.
I propose the article should be retitled ‘Subjective probability’, in which ‘probability’ is interpreted as ‘strength of belief in the truth of a proposition’, and accordingly purged of all the confusing stuff about other conceptions. I note that a Wikipedia search on the term subjective probability already redirects to this article, whereas searching on ‘Objective probability’ gives no article on that topic.
I have no idea what the article’s current definition of ‘Bayesian probability’ as “a state of knowledge about a proposition” means. Does anybody ?
I note that the Wikipedia ‘Bayesian inference’ article currently starts by mistakenly defining Bayesian inference as concerned with “the probability that a hypothesis may be true”, thus mistakenly restricting Bayesian inference to objective probability, whereas Bayesian inference is also used in subjective probability in which ‘probability’ is ‘strength of subjective belief that a hypothesis is true’ rather than ‘the probability that it is true’.
--Logicus (talk) 18:11, 5 March 2008 (UTC)
I did not read your dense prose completely but (a) I am in favor of renaming as you suggest and (b) probability as a state of knowledge is straight out of Jaynes. Tomixdf (talk) 18:47, 5 March 2008 (UTC)
"Epistemic probability" is what the article is about. I'm not sure that "Subjective epistemic probability" is right, because there are some - and historically have been even more - who take the view that epistemic probabilities can be assigned objectively.
However, in the real world, "Epistemic probability" is overwhelmingly called "Bayesian probability", and associated with the Bayesian counter-revolution against Frequentism and Frequentist statistics.
A key WP principle is to name articles in accordance with the most familar usage. In this case, that is "Bayesian probability", being the view of probability adopted by modern-day Bayesians. Jheald (talk) 23:13, 6 March 2008 (UTC)


OK, but then surely something crucial as Cox's axioms or Jaynes' view of probability as measuring a state of knowledge should be discussed in the article. The current mainstream view on Bayesian probability is distinctly objective/logical, a la Jaynes (see for example Bishop's recent hugely popular text book). In pratice, that is reflected in the use of Maxent priors, priors based on invariance etc. There's nothing subjective about it, at least not in principle. If this article is to be about Bayesisan probability in general, than I do not understand why it should focus solely on this so-called 'subjective probability', which has now become a minority view (but of course it can be dicussed and mentioned!). Tomixdf (talk) 07:21, 7 March 2008 (UTC)

I don't have time to enter this debate again, but I feel the need to warn everyone that Logicus has, for a while now, been trying to add densely written paragraphs which turn out to make little sense when you take the time to understand them thoroughly. Go back and look at the modifications he made last year if you don't believe me. Also have a look at the discussions above.--BenE (talk) 15:54, 7 March 2008 (UTC)

Yes, I've also noticed something seems to be wrong. So what do we do? We can either (a) give up and rename the page to something like "Subjective epistemic probability" (which will make the problem disappear into obscurity) or (b) try to get to a decent history and overview of the different Bayesian schools (Keynes, Jaynes & Jeffreys, de Finetti,...) into the article. The current state of the article is unacceptable for such an important topic. Note that disruptive users can be reported to the administrators (had some very good experiences with that way of getting rid of disruptive edits recently). Tomixdf (talk) 19:32, 7 March 2008 (UTC)

The section "The Controversy between Bayesian and Frequentist Probability" is IMO very poor. I'm planning to delete it. Opinions? Tomixdf (talk) 19:58, 7 March 2008 (UTC)

Logicus to Tomixdf: Jaynes is an objectivist, not a subjectivist. Having agreed this article should be about subjective probabilism, you have now launched into defining it as objectivist using a Jaynes 'definition'. This will cause even deeper confusion. More later ... --Logicus (talk) 15:46, 16 March 2008 (UTC)
We've agreed no such thing. This article is about epistemic or personal probability, and should cover both those who believe that such personal probabilities can be arrived at objectively, and those who don't. Jheald (talk) 16:02, 16 March 2008 (UTC)
Logicus to Jheald: I did not say that you and I did agree any such thing, but rather in my message addressed to Tomixdf I said that Tomixdf did agree that, for as he said on 5 March above:
“At one point Logicus mentions "the article's definition of Bayesian probability as subjective epistemic probabilism". It seems the article should be renamed to reflect what it is actually about then (ie. subjective epistemic probabilism, or so it seems).”
and also
“…but (a) I am in favor of renaming as you [i.e. Logicus] suggest.”
May I respectfully suggest you should perhaps consider reviewing whether you are sufficiently functionally literate in English to be attempting editing Wikipedia articles, let alone arrogantly asserting Humpty Dumpty style what articles on the philosophy of probability are about ? I merely pointed out that given the article's previous definition of Bayesian probability as subjective probability, it should be renamed as such. --Logicus (talk) 19:43, 22 March 2008 (UTC)


Indeed, I strongly agree with Jheald. Moreover, the article should also make clear that the predominant view of Bayesian probability today is Jeffreys/Jaynes's objectivist view. Tomixdf (talk) 16:10, 16 March 2008 (UTC)
Erm. I think you'll find that by the end of Jaynes's life, he was backing away from the objectivist view. Yes, you may often be able to derive a prior probability distribution from a transformation group; but the choice of that transformation group my nevertheless itself be personal, and therefore subjective. I believe the article's summary is correct, that most present-day Bayesians are content to accept that there is a subjective element; though some are more objectivist.
Jaynes' backing away from the objectivist/logical view? His magnum opus "The logic of science" was the last thing he worked on (he died while working on it), and his views in that book are strongly objectivist/logical, as far as I know (I am reading it at the moment). Could you point me to specific references for this? (I'm asking out of interest!) Also, see page 655 for his quite critical views on de Finetti. Tomixdf (talk) 17:18, 16 March 2008 (UTC)
In any case, a good Bayesian should stress-test their conclusions against the effects of different priors. Jheald (talk) 16:35, 16 March 2008 (UTC)


Logicus to Tomixdf: The current introduction to this article on the PHILOSOPHY of probabilIty after your recent re-editing of it is now as follows:
"Bayesian probability interprets the concept of probability as 'a measure of a state of knowledge' [1]. Broadly speaking, there are two views on Bayesian probability that interpret the 'state of knowledge' concept in different ways. For the objectivist school, the rules of Bayesian statistics can be justified by desiderata of rationality and consistency and interpreted as an extension of Aristotelian logic[2][3]. For the subjectivist school, the state of knowledge corresponds to a 'personal belief' [4]. Most modern machine learning methods are based on objectivist Bayesian principles [5]."
But this has introduced even greater conceptual confusion as a result of your edits, and whereby it increasingly seems that you may lack the requisite learning and philosophical and logical competence in the subject matter here, namely the PHILOSOPHY of probability in the interpretation of the term 'probability', to contribute to its clarification at this stage in your education in that subject. It would seem maybe you are some kind of statistician who is only just now learning about the philosophy of probability, as suggested by the fact that you are only just now reading Jaynes' work on objective probability as the logic of science, and thus can hardly have had time to make any serious critical philosophical assesment of it and why Jaynes' objectivism is widely and rightly regarded by philosophers of science as puerile nonsense. (For example, you should consult the 1993 Howson & Urbach book listed for the elementary subjectivist demolition of Jaynes's objectivism as pseudo-objective.)
Here I just briefly point out the conceptual confusions now arising from your decision not to make this article about subjective probability contrary to what you had apparently agreed on 5 March, but rather about what is an utterly spurious non-subject, 'Bayesian probability', such as ERosa cogently pointed out. Overall the mystery you have now even more confusingly reproduced is that of what on earth is distinctly 'Bayesian probability', which the article now claims takes two forms, namely objective and subjective ? So what then is non-Bayesian probability, pray ? The real answer is of course that there is no such thing as specifically Bayesian probability, inasmuch as ALL conditional probability calculus uses Bayes Theorem.
Your first sentence now defines 'Bayesian probability' in terms of an objectivist Jaynes characterisation as 'a measure of a state of knowledge', typical incoherent Jaynes-twaddle presumably meaning rather 'a degree of knowledge', and which is objectivist inasmuch as 'knowledge' is taken to be justified true statements whose truth-value is independent of the knowing subject and determined objectively by the real world. So a statement whose truth is only half justified would have probability 0.5 on this view.
Your second sentence then tells us there are two different interpretations of the 'state of knowledge' concept within this illusory 'Bayesian probability'. But the remaining 3 sentences collapse into nonsense or irrelevance in explicating these alleged two different interpretations. The second of these remaining sentences tells us that in subjectivist probability " 'state of knowledge' corresponds to a 'personal belief' ", which is presumably an illiterate attempt to say " 'state of knowledge' is interpreted to mean 'strength of personal belief that a statement is true' ". But of course the latter is not in itself a state of knowledge at all, as illustrated by the strong belief of Russell's turkey that the farmer would feed it as Xmas rather than wring its neck at Xmas.
But the preceding sentence totally fails to tell us how objectivism interprets 'state of knowledge' in comparison, and instead irrelevantly reports the philosophically foolish nonsense that objectivists believe in, namely that 'the rules of Bayesian statistics can be justified by desiderata of rationality and consistency', a view that any capable philosophy of science undergraduate should easily be able to demolish.
And what machine learning methods happen to be is surely quite irrelevant to explicating different concepts of 'probability'.
So what is the objectivist interpretation of 'state of knowledge' ? One problem here is that you are likely end up resorting to the classical definition of probability, widely regarded as non-Bayesan or pre Bayesian', wrongly or rightly, or some such mumbo-jumbo as ‘rational belief in the degree of certainty in an event’. Or else end up misdefining probability as verisimilitude, that is, as a degree of truth and thus a degree of knowledge.
As it is, the introduction is now in an even worse state of illogical obscurantist confusion.
I propose it should at least be replaced to what it was before you started meddling with it, and the article retitled 'subjective probability' and re-edited in line with this. In keeping with that change, there should also be another article created on 'Objective probability', and the pedagogically confusing conceptual nonsense of 'Bayesian probability', an empty category, committed to the flames of meaningless metaphysics where it belongs.
May I recommend you read the listed literature for this article, and especially the works by such as Howson & Urbach and by Gillies if you are trying to educate yourself about the philosophy of probability and also about probabilist philosophy of science. For what it may be worth, my simple opinion is that whilst subjectivism may be the philosophically most tenable interpretation of the concept ‘probability’, on the other hand the logic of science, contra Jaynes and also contra the subjectivists, is definitely not probabilist, i.e. a subjectivist philosophy of probability but an anti-probabilist philosophy of science. But the purpose of the article to report viewpoints in the literature and their problems.--Logicus (talk) 20:17, 22 March 2008 (UTC)

The first sentence currently reads, "Bayesian probability interprets the concept of probability as 'a measure of a state of knowledge' [1]." This does not define what Bayesian probability is, but what it does. Consider the readable definition under the Thomas Bayes article, "Bayesian probability is the name given to several related interpretations of probability, which have in common the notion of probability as something like a partial belief, rather than a frequency."

A concise, straightforward first sentence will help make this article more accessible. Kevin.j.hutchison (talk) 00:42, 1 May 2008 (UTC)

[edit] The Controversy between Bayesian and Frequentist Probability

This section is such a mess, I'm cutting it to here, so we can discuss it.

The theory of statistics and probability using frequency probability was developed by R.A. Fisher, Egon Pearson and Jerzy Neyman during the first half of the 20th century. A. N. Kolmogorov also used frequency probability to lay the mathematical foundation of probability in measure theory via the Lebesgue integral in Foundations of the Theory of Probability (1933). Savage, Koopman, Abraham Wald and others have developed Bayesian probability since 1950.
Note: Abraham Wald was not a Bayesian. He was a frequentist. But he showed that in decision-theory terms, any frequentist inference rule would be "inadmissable" - ie demonstrably sub-optimal - unless it was equivalent to a Bayes rule. Jheald (talk) 18:42, 19 March 2008 (UTC)
The epistemological difference between Bayesian and Frequentist interpretations of probability has no important consequences in statistical practice.[dubious – discuss] But in regards to the use of the term "Bayesian" in a mathematical sense, the practical difference is whether prior information is included in a calculation of a posteriori probability. Some Bayesians claim that non-Bayesian sampling statistics assume that one knew nothing of the thing being sampled prior to the sampling. But this claim is not true. Of course, the assumption is almost never true in the real world.[1] A frequentistic approach would be to consider the distributions of random variables representing all past sampling efforts as well as the present effort - summary statistics can be used instead of complete samples and the contributions could be represented by likelihood functions in parallel to the Bayesian approach. Bayesian analysis simply allows prior probabilities to be taken into account when interpreting observations. This may appear simpler than the frequentist approach but is essentially equivalent when "prior information" reflects past samples. The Bayesian approach has the hidden danger that the apparent simplicity of the formulae used means that an important underlying assumption is forgotten, specifically that the random quantities in the sample being analysed should be statistically independent of the information summarised by the prior distribution.
In fact, Bayesian analysis can even use prior information that was based on other statistical sampling methods and need not be associated with subjective methods at all. Furthermore, it can be shown that over a large number of trials, even a subjective "calibrated probability assessment" will agree with observed distributions (i.e. an "80% certain prediction" will be right 80% of the time).[2] This result means that Bayesian analysis, even when it is based on subjective probabilities, will agree with the frequentist's approach.[dubious – discuss] Finally, it must be noted that Bayes Theorem itself is mathematically proven and is not at all subjective. It is derived from axiomatic elements of probability theory and makes no reference to whether prior knowledge is subjective or based on a large number of observations. These points together largely blunt any practical difference between the two philosophical positions when applied to real statistical problems[dubious – discuss] and, in reality, it is not uncommon for statisticians to use both. However, there continues to be confusion about the use of the term Bayesian in regards to the epistemological position, and its practical use in statistics.[dubious – discuss]
The cluelessness here is simply extraordinary.
Firstly, probably the most important advantage of Bayesian methods in practise has nothing to do with priors. It is that Bayesian methods allow marginalization over the values of nuisance parameters. This opens up huge possibilities for realistic hierarchical models simply not available in Frequentist methods.
Secondly, these paragraphs discuss Bayesian methods as if the only thing they are relevant for is pseudocounts. Again, this is way off the mark. Bayesian methods give a systematic and probabilistically coherent method for setting up inferences of any quantities in a generative model. Compare that to Frequentist methods where often there would be no clue as to how to set up an appropriate model.
Intricate hierarchical models, with marginalization over nuisance parameters, are bread-and-butter for Bayesian MCMC engines. How would you even start to estimate such models in a Frequentist way?
Thirdly, "no important consequences in statistical practice" -- this is very blinkered nonsense. There are ample examples where Frequentist methods give either physically impossible answers, or (as per the example discussed in Confidence Limits section above) where Frequentist answers are hugely misleading. Compare that to a Bayesian answer, which really does give the best estimate of the combined posterior probability distribution, given the data and the model it's been fed.
Fourthly, "the apparent simplicity of the formulae used means that an important underlying assumption is forgotten". If you start from the generative model and Bayes theorem, nothing is forgotten. If you happen to have mangled some of your data into summary statistics (probably not the best idea, if it can be helped), and there are non-trivial dependencies, then the full Bayesian approach needs must include those dependencies, and the inference will have to take them into account. Starting from first principles: a lot more transparent than the black-art cookbook that is Frequentist statistics. Jheald (talk) 18:42, 19 March 2008 (UTC)
Well yes, but some versions of "Bayesian" stuff suffer from the same cookbook effect. For example, the presently remaining part of the article says "Bayes's Theorem is explicitly or implicitly used to update the strength of prior scientific beliefs" without implying this may be difficult or requiring particular thought. Melcombe (talk) 10:56, 25 March 2008 (UTC)
More criticisms could without doubt be piled up about the section above (eg the extraordinary claim about "calibrated probability assessments" -- rather depends who you ask, and what they're estimating, I suspect). But the more fundamental point is this: If we're going to have a section on the "controversy" (and we should), it should focus on the charges the leading participants on each side actually levelled, and it should be referenced (WP:RS). Random unsourced claims and attestations are not good enough.
To clear the ground to allow such a section to be built, it seems appropriate to cut all the existing material to here, and simply start again. Jheald (talk) 18:48, 19 March 2008 (UTC)
My thoughts are:
(i) perhaps it would be better to divert attention to some other existing article, possibly somewhere under "inference", with only a very brief discussion in the present article;
(ii) perhaps it would be good to start from where Bayesian and Frequentist ideas are in most agreement, specifically with the likelihood function.
Melcombe (talk) 10:56, 25 March 2008 (UTC)

I came to this article hoping to find enlightenment on what this Bayesian vs. Frequentist debate was about. The term 'frequentist' appears a couple times in the article, but without explanation apart from a link to an outside webpage. And this discussion page is full of debate, but since I don't know yet what the (alleged?) difference is, I can't make heads or tails of the debate.

Can someone please add an explanation to the article on what this debate is about? Or else start a new article on the differences? Mcswell (talk) 19:43, 25 April 2008 (UTC)

[edit] The word "probability"

"Probability" is a precise term in mathematics, isn't it? Though I'm not competent to do it, it seems to me that several occurences of the word in this article ought to be replaced by "credibility," "likelihood," or "cogency," and that the semantic complications need further elucidation. The article suggests that there are disputes over what the word "probability" ought to mean, rather than advances in understanding which sort of problems may be amenable to analysis and how to solve them. Unfree (talk) 17:55, 26 April 2008 (UTC)

[edit] Editing long lines

I've decided to alter somebody else's contribution to this discussion because it contains some very long lines which make it hard to read the page and navigate through it. Unfree (talk) 20:08, 26 April 2008 (UTC)

No, I haven't done it, but the problem appears to occur because of the presence of

style="white-space: nowrap;"

in the html source. I don't know how to fix it, and hope somebody else will. Unfree (talk) 20:17, 26 April 2008 (UTC)