Talk:Margin of error

From Wikipedia, the free encyclopedia

Former featured article Margin of error is a former featured article. Please see the links under Article milestones below for its original nomination page (for older articles, check the nomination archive) and why it was removed.
Main Page trophy This article appeared on Wikipedia's Main Page as Today's featured article on October 19, 2004.
WikiProject Spoken Wikipedia

User:Athf1234 is currently recording or editing an audio version of this article.
See WikiProject Spoken Wikipedia for further information.

This notice is dated 2006-01-16. If a significant amount of time has passed,
please check with User:Athf1234 to see if he or she is still working before removing this message.
Peer review This Math article has been selected for Version 0.5 and subsequent release versions of Wikipedia. It has been rated B-Class on the assessment scale (comments).
WikiProject Mathematics
Mathematics Portal
This article is within the scope of WikiProject Mathematics, which collaborates on articles related to mathematics.
Mathematics rating
A-BB+ Class Top Priority
 Field: Probability and statistics
One of the 500 most frequently viewed mathematics articles.

This article is within the scope of WikiProject Statistics, which collaborates to improve Wikipedia's coverage of statistics. If you would like to participate, please visit the project page.

Contents

[edit] Concerns

There are some concerns about this article's depiction of the margin of error as a statistic used specifically for polling. There is a discussion below about whether the margin of error is synonymous with the confidence interval and whether, as such, it should be more broadly defined. Fadethree 8:36, 19 Oct 2004 (UTC)

[edit] TeX (moved to Wikibooks)

Near the end of the section on comparing percentages we see this:

= normdist((A1 − B1),0,A2,1)

This seems to be a dangling phrase, not part of any sentence; it needs to be clarified. I'm guessing it mean the cumulative distribution function of a specified normal distribution; it so, there are clearer ways of saying that. Michael Hardy 00:22, 4 Oct 2004 (UTC)

  • Hi, Michael. Thank you for the valuable suggestions. I should have known better about the capitalization and spacing issues; thanks for the corrections. Regarding the do-it-yourself section, my intention was to allow the user to simply copy and paste the line of code into a Microsoft Excel spreadsheet so that they could calculate the probability that Kerry (or anybody) is leading given the information from a poll. I reverted it back for two reasons, 1) to allow the cut and paste and 2) to match the look and feel of Microsoft Excel, which uses A1 B2 C3 notation without subscripts. I changed some things around to make it clear that I was working in the context of Excel. Let me know what you think; I am happy to revert it back given different intentions. Best, Andrew (Fadethree) 05:35, 4 Oct 2004 (UTC).

In section "Comparing percentages: the probability of leading", is there any chance we could change " use a program like Microsoft Excel " to " use a spreadsheet program "? In my humble opinion, this would improve the Neutral Point of View... Charm 18:17, Oct 21, 2004 (UTC)

[edit] "poll"

If I'm not going completely mad, isn't use of the word poll a little inaccurate? Shouldn't it be a sample or sample survey? Or at least it is according to my textbook! Please advise... I'd hate to incorrectly edit a front page article

Also, I would have said, for example, 5% significance level rather than 95% confidence level. Does this equate to the same thing, or is a difference in terminology across the Big Pond. If so, should both definitions be included?

How is poll inaccurate? We've been having a discussion below about assumptions that the margin of error is synonymous with the confidence interval. Is this your assumption? If so, note that this article defines the margin of error as a quite specific confidence interval; they are not equivalent. As to your second point, I am fairly certain that 95% confidence level is synonymous with 5% significance level. I would think that confidence is not as apt to be misconstrued as "significance." This talk section has been largely concerned with semantics, and while semantics are nontrivial, I would argue that the dominant audience of this article is people who are trying to understand the margin of error without a statistics background. I would like to err on the side of parsimony. Your thoughts? Fadethree 21:27, 19 Oct 2004 (UTC)
Wow! I will admit that I'm being a little pedantic here, and certainly didn't mean to rub someone up the wrong way, but found during my lessons in stats terminology was very important. Point taken on both counts - the pedant in me is perhaps more needed on the statistics page! Cheers --Mark Lewis 14:34, 20 Oct 2004 (UTC)

[edit] Picture idea

If this article becomes featured, a picture should be added, in fact, one probably should anyways (:. How about some sort of graphic depicting the margin of error in the provided kerry/bush example? —siroχo 09:14, Oct 4, 2004 (UTC)

[edit] Re: Level of "Confidence"

(This comment copied from Fadethree's talk page:)

Regarding my comment on the FAC page. I appreciate the fine work you've done to get the article to this point. I appreciate your explanation, and I noticed you gave the alternate forms indirectly. I guess the pedant in me simply cringes at leading with an expression which is not fully qualified. Perhaps if in introducing the equation you could mention the 99% confidence interval (though I admit this could be ugly too). I would be happier if the equation was generalized with the proper variable in the numerator and a notation that "for 99% confidence, (numerator) is 1.27".

Also, several references I've seen (an online one is the iSix Sigma site claim that a 95% confidence interval is more common, which matches with my experience also. This may be a more serious issue than the form of the equation. Jgm 13:19, 6 Oct 2004 (UTC)

First of all, let me say that I think that you're right that this should be made clear from the outset, and after this comment I'll try to rearrange the definition to get the "disclaimer" moved higher up than the last line. It might not be as pretty, but I do think it's worth it for now. I may give up and just expand the last line and bold-face some issues I talk about below, but I'll try to move it up first.
To reply to the substantive issues, you're right that 95% confidence is by far and away the standard for "significance" in most of the sciences (though it is usually expressed as p<.05 or some such). In recent media polls, however, the standard seems to be, by far and away, a 99% margin of error. (edit: counterexample the AP poll ([link]).) This can easily be checked in any poll that reports both the sample size and the margin of error (if 1000 is 4% and 1500ish is 3% say). There are a few points to be made here about reporting widely misunderstood topics.
  1. First, no one says, "99% margin of error" or 95%, or anything like it. In other words, there seems to be an illusory consensus here about what the confidence should be. Wikipedia is bound to report what this consensus is, but it is also bound to educate people if this consensus is misleading or damaging. This is why I agree with you that we should make this more clear.
  2. Second, this is an interesting interpretation of the "mission" of Wikipedia. It is bound to report "facts," but what if the way that people understand a topic is fundamentally flawed and potentially misleading? Should it also be bound, then, to educate?
  3. Third, who is the intended audience here? Is it curious members of the voting public who read a poll and want to know what the margin of error is all about? Or is it people who are knowledgeable consumers of statistics and want to use the margin of error for their own social science (or other) work? The opening paragraph should try to aim for as broad a target audience as possible, but the distribution of prior knowledge here seems vast. These are just some thoughts. Fadethree 14:41, 6 Oct 2004 (UTC)

[edit] Re: Should a real or hypothetical example be used?

(This comment copied from the FAC page)

We should use a made up and neutral example, so we need not use disclaimers like "It should be clear that the choice of poll and who is leading is irrelevant to the presentation of the concepts." These disclaimers are ugly in the flow of a well-written article. ✏ Sverdrup 10:14, 6 Oct 2004 (UTC)

I guess we can frame this as an issue of accessibility. It is a great point that Wikipedia articles should be as general and neutral as possible. This suggests stripping the running example of as much context as possible. I have spent some time teaching, and a successful strategy I've used is meeting the audience (students or web surfers) at the level of their experience. It is much easier to learn a topic when you can relate to it, and this is why I've included an example that I think most people who hear "margin of error" can relate to. If this article does become featured, I think that this would be a great "hook." However, there is also a larger point that, well, this election will be over soon, and this article will live on well past Bush vs. Kerry. A neutral example would probably be applicable for a longer time (though of course it could be updated). I'd like to solicit some other people's opinions (including Sverdup's response; thanks for your comment!). Fadethree 14:41, 6 Oct 2004 (UTC)

Personally I would say made up examples tend to lose people too easily, current examples tend to be contentious and require disclaimers, maybe a real but slightly older example might work as a compromise to avoid either issue?

I think you're right. Creating hypothetical names that are easy to distinguish would probably be an easy way to keep the situation relevant. We could use "Larry Leader" and "Pete Pariah" for the candidates.

[edit] questions

you say assume a 99% level if unspecified to be conservative. 1st, my impression is that 95% is typically the default. 2nd, in that case the 99% assumption understates the variance and thus is actually unconservative, or am i missing something here?

second, why use the constant 1.29 in the 1st example? that is correct for a point estimate of 1/2. For any other point estimate, the constant is wrong. (i do realize you give the correct formula later).

your probability of leading formula assumes perfect negative correlation. i assume here that you mean q = 1-p; why then not make the substitution p - q = p - (1-p) = 2 p -1; so variance = 4*p*(1-p)/N. that reads a lot simpler than the complicated formula you provide.

i generally like this article, and agree that it is a subject woefully misunderstood by the public & press. often people think anything within the margin of error is equally likely, not true of course. it might be a nice example to show how 2 indepent polls can be combined to give a new estimate, with a smaller margin of error. Wolfman 03:50, 7 Oct 2004 (UTC)

  • These are very illuminating questions. Thanks for checking in.
  1. As I note to Jgm above, 95% is standard for most applications in the sciences, but 99% appears to be the standard for most polls reported by the media. As I note there, you can confirm this because polls that have a sample around 1,000 say that their margin of error is 4% (it would be 3% if a 95% confidence interval were used). As I also said to Jgm, this ambiguity (no one specifies what confidence level the margin of error should be) adds to misinterpretations about the statistic. (edit: though it looks like the AP is using the 95% ([link]). I think, though, that as long as some respected pollsters are using 99%, all of them should unless they report the level of confidence.)
  2. I think that you're missing something, but it's not an uncommon slip. Remember that in order to have higher "confidence," the confidence band must be larger. To be sure that you have a confidence band of 99% for a reported 50%, therefore, the margin of error must be larger. This is more conservative because it suggests that the poll has more potential error.
  3. You hit the nail on the head with this question. As you say, for any other point estimate, the constant is wrong. Yet this is the definition of the margin of error. The key thing to remember is that the margin of error is not the confidence interval of any percentage, it is the confidence interval placed at a particular reference point for the purpose of comparing polls.
  4. Another great question. You've just proven that what could be called the margin of error of the difference is twice the margin of error. It is a useful result for approximation. However, perfect negative correlation does not necessitate q=1-p, just that any person who switches votes switches to the other candidate and no other. Also, if we assume that q=1-p, we assume that q+p=1, and this is rarely true in most polls. Note that if it were true, we wouldn't need the standard error of the difference at all, we could just calculate the probability that p>50%.
  5. I have reservations about combining polls to reduce the margin of error because this requires an assumption that they are both random samples of the same population. This is a tenuous assumption for one poll let alone two, and often the discrepancies between polls tell us not about the population but about the respective sampling procedures of the pollsters. Still, it might be a valuable mathematical exercise, and I would be happy to have you add that section should you summarize these kinds of reservations. Thanks again for the insightful questions! Best, Fadethree 18:43, 7 Oct 2004.

[edit] Is 95% or 99% more common/conservative?

A [Zogby] poll came out today that used the 95% confidence interval, so I'd like to revise my earlier claim that "99% is by far and away the standard..." I should have looked at more polls before making that claim. However, I do not think that this refutes the point that I made in the opening paragraph, "...if the report does not state the confidence level... the 99% equation should be used." As long as some polls are using the 99% equation, they are using a higher standard (their margin of error will be larger for the same sample size). Therefore, the burden of fair reporting falls more squarely on the shoulders of polls using a lower standard. Overall, it is clear that the confidence level should always be reported, because the margin of error is not uniquely defined. This is my opinion as of now; let me know your thoughts. Fadethree 16:57, 11 Oct 2004.

Suppose a poll says the margin of error is 3% without specifiying the confidence level (which is really 95%). Suppose I am personally interested in knowing the 95% confidence level. If I assume that the reported level is 99%, then I would infer that the true 95% level is maybe 2%. In that case, I am overly confident in the polls accuracy, so assuming the 99% level is not conservative.
On the other hand, suppose 3% is actually the 99% margin of error, but I wrongly assume it is the 95% level. If I then want to know the 99% level, I would guess maybe 4%. That is an overstatement of the true margin of error, and thus conservative in that it understates the accuracy.
So, unless I have just got my head turned around wrong on this, it seems to me that the 95% assumption is the conservative one. And thus the safer level to use if the poll does not report the correct confidence level. Wolfman 04:03, 12 Oct 2004 (UTC)
I will try to reframe your argument in a way that is hopefully accurate and hopefully illuminating. There are three variables regarding the confidence level. A) What you assume it is, B) What it actually is, and C) What you want to calculate. Your argument reduces to the following premises: If A < B then C will be overestimated and conservative. If A > B then C will be underestimated and not conservative. You concluded correctly that it is conservative to assume the confidence level is less than it is ("...it seems to me that the 95% assumption is the conservative one."). But this tells us what is more conservative to assume, not what is more conservative to report. As you can see, if we want to be as conservative as possible, we want to minimize A and maximize B. As I noted in the introduction, the 99% equation should be used (i.e. reported) so that no matter what the readers assume, they will be conservative.
Of course, ideally, A = B so that C will always be right. This is one of the reasons why I advocate reporting the confidence level. Even if the confidence level is not reported, the formula for the margin of error can be inverted and solved for the numerator, which we can match to the confidence level.
Now, you could make an argument that this article suggests that the 99% confidence level should be assumed, and though I don't think it does, this would be a bad thing. Any tips that you suggest to make this more clear would be welcome. Thanks again for your question! Fadethree 6:23, 12 Oct 2004 (UTC)
From your reply, I think we agree on the basic principles. This is the line of the article that throws me: "If a report does not state the confidence level of the margin of error, the 99 percent equation should be used to prevent readers from underestimating the potential variance of poll results." I now think what you mean by this sentence is: if an author chooses not to report the level, he should use the 99% figure. I previously read it as: if an author has not reported the level, the reader should assume the reported level is the 99% figure. It's the passive voice "should be used" that creates the ambiguity. Wolfman 06:41, 12 Oct 2004 (UTC)
Gotcha. I see how that sentence is vague. I have changed it to, "If an article does not state the confidence level of the margin of error, it should report the margin of error at 99 percent confidence to prevent readers from underestimating the potential variance of poll results." I am still wondering if I should be more clear, but I'll sleep on this version for now. This was helpful; thank you. Fadethree 6:50, 12 Oct 2004 (UTC)
I find your revised wording clear. And, kudos on a fine article. Wolfman 07:01, 12 Oct 2004 (UTC)

[edit] Revisions

I have done a survey of polls, and I have concluded that 95% is certainly common enough to be designated "common." I have reorganized the opening to be more concise and take note of the different standards that are out there, and I have added formulas for the 95% (and 90%) confidence levels as well. I think this makes for a much more well-rounded article, and I'd like to thank Wolfman and Jgm for highlighting this issue for me. Best, Fadethree 21:48, 13 Oct 2004 (UTC)

[edit] Levels of confidence

It says:

The margin of error can be calculated directly from the sample size (the number of poll respondents) and may be reported at three different levels of confidence.

Surely it can be reported at far more than just three levels. Nothing stops a statistician from reporting a 98% or 97% margin of error. Maybe what is meant is that those three levels are frequently mentioned? If so, maybe this should be rephrased. Michael Hardy 01:14, 19 Oct 2004 (UTC)

Absolutely right, Michael. I have clarified this as follows, "...and is commonly reported at one of three different levels of confidence. The 99 percent level is the most conservative, the 95 percent level is the most widespread, and the 90 percent level is rarely used." I wonder if I should be more explicit... feel free to try another wording. Thanks also for your helpful edits to the opening. Best, Fadethree 01:47, 19 Oct 2004 (UTC)

[edit] Margin of Error not the only posible poll error

Should this article have a section about the need for the poll to have a truely random sample for the result the valid. ie. the idea that doing the Bush/Kerry poll by sampling everybody who works at the White House would not give an acurate result. I know this isn't strictly "margin of error", but some people won't realise this and (maybe) it whould be pointed out Steven jones 01:41, 19 Oct 2004 (UTC)

Thanks, Steven. There is a note about this under the caveats section, "...due to its unfortunate name (it neither establishes a "margin" nor is the whole of "error")..." and under the arguments against... section, "Perhaps most importantly, there are many different sources of error in polling..." In my first drafts for the article, I placed the misconceptions and overinterpretations at the center of the article, but many people noted, correctly, I think, that the article should center first on what the margin of error is and second on the overinterpretations. Your thoughts? Fadethree 02:00, 19 Oct 2004 (UTC)

Last paragraph of the Explanation section ("If the exact confidence intervals are used, then the margin of error takes into account both sampling error and non-sampling error") seems clearly inconsistent with Basic Concept section ("However, the margin of error only accounts for random sampling error, so it is blind to systematic errors..."). The Basic Concept section is correct, so I suggest deleting most of the last paragraph of Explanation section so that it reads: "The margin of error does not represent other potential sources of error or bias such as a non-representative sample-design, poorly phrased questions, people lying or refusing to respond, the exclusion of people who could not be contacted, or miscounts and miscalculations." Charlie nz 10:28, 23 August 2007 (UTC)

[edit] population size

It's a common intuitive assumption that the ratio of (same size) / (population size) impacts the margin of error at a given confidence level: that is, a sample size of 1000 out of a population of 10,000,000 is more accurate than 1000 out of 100,000,000. This isn't the case for "large enough" populations, but isn't discussed in this article. I don't know the details well enough to discuss it precisely, but it's a frequent misconception, so should probably be mentioned. --Delirium 02:31, Oct 19, 2004 (UTC)

Yes, that is a very common misconception. I think it may be more appropriate for the sampling (statistics) page, however. The margin of error is, of course, a statistic that is derived from sampling theory, however the misconception applies to all sampling theory, not just the margin of error. I'd be happy to see it added at that page, of course. Does this make sense? Fadethree 03:00, Oct 19, 2004 (UTC)
I took the liberty of adding a section discussing this (in somewhat informal terms), at the margin-of-error page, but perhaps it should eventually be moved to the sampling page; I'm agnostic about this. Terry 18:15, 21 Oct 2004 (UTC)
I appreciate the contribution, Terry. I do think that the topic is an imperfect fit here and that it makes more sense at the sampling page. Still, if this is the one-stop place for people who want to understand the topic, your addition will certainly be appreciated here. Wherever it ends up, I think it will be very helpful. Fadethree 10:26, 23 Oct 2004 (UTC)

[edit] Other uses of margins of error

I don't know if there should be more information appended to the main article or setup as separate pages, but margins of error are used in other fields for related reasons. For example:

  1. In many fields that use digtal computation the margin of error is the sum of all the possible sources of error including the quantization error, sampling error, etc. - giving an area which a given measured value could represent. As such this shares a simular meaning for margin of error, but is applied to a completely different problem.
  2. In some fields of engineering a margin of error is also known as a safety margin and is used an index indicating the amount beyond what is strictly necessary. For example, the margin of error used when building a bridge may be the amount of additional strength that is provided beyond what is strictly necessary based on the mathematical calulations and this is used as a precaution to ensure that the structure doesn't fail and under non-ideal conditions.

Fanboy 06:00, Oct 19, 2004 (UTC)

This is very interesting, Fanboy. I had never heard of these uses of the term. In some senses, the phrase's many meanings speaks to the ambiguity of the phrase "margin of error," and makes the common misinterpretations of the term less surprising. I think that these usages are sufficiently independent to warrant separate pages. I'm not sure if we can call this current page Margin of error (statistics), since your descriptions are also statistical in nature. Perhaps Margin of error (polling). I wouldn't want to make that move until other pages about the Margin of error were up. Thoughts? Fadethree 06:28, 19 Oct 2004 (UTC)
The common thread between all varients of the phrase "Margin of Error" is that the phrase is used to indicate a given range that may be attributed to errors in a calculation or measurment. In popular culture (particularly around the time of political events) this phrase becomes part of the common vernacular because of polls - which is why this page probably was formulated the way it is and why I'm not sure about adding another page. I actually think that the way Trollminator had changed things (which was rolled back) might be a good thing to put back in place as it provides a slightly more accurate definition of Margin of Error - but it still uses polls as primary example of margin of error since this is the primary usage by the general public. Fanboy 06:46, Oct 19, 2004 (UTC)
There are some interesting semantic points here. You probably noted my comment at User talk:Trollminator, and as I note there and in the article, the margin of error (that the article discusses) is a manipulation of the standard error of measurement that is used to compare polls. However, I do not believe that we should confuse the margin of error with the standard error of measurement, which is what I think Trollminator and your definitions are beginning to do. I think that the definition that you note, "the phrase is used to indicate a given range that may be attributed to errors in a calculation or measurement" is already well defined as a confidence interval and should be left at that.
In a sense, defining the "margin of error" as a statistic for polling is a method of triage. The term is simply misleading as a expression of error; as you know, the notion of a "margin" for sampling error is illusory; 95% or 99% are arbitrary standards that have little to do with the complex decisions that these statistics purport to inform. So, I would argue, conflating the "margin of error" with the "standard error" or standard error-based statistics is to perpetuate confusion and misinterpretation for potentially high-stakes decisions (as examples in the article demonstrate).
I think this is an important conversation to have, and I hope I am making my points clear. Please let me know if I am making any sense. Best, Fadethree 07:19, 19 Oct 2004 (UTC)
The problem with this topic is that because "margin of error" in statistics is a measure of the uncertainty in an estimate of a parameter it gets used in different ways by different people who are using the same statistics base.
The errors in polling almost exclusively relate to uncertainty in getting a grasp of the topic at hand - due to the complexity of a situation or modelling the entire population on a smaller potentially non-representative group. The "margin of error" is used to indicate that although you can measure responses perfectly (since in a poll usually everything is a discreet event; yes/no, blue candidate/red candidate/green candidate) there is still some uncertainty as to whether the parameter you are estimating is properly serviced by the question posed.
The "margin of error" definition used in my first point above is a calculated range based on the knowledge that when measuring something digitally you're not sure exactly what you are measuring. This is sometimes used in designing a system as a tolerance of what may be allowed or what may be measured. Try doing a Google search for "margin of error GPS" and you will see another common use of the phrase peppered throughout literature related to mapping coordinates using traditional and GPS systems. In my second point, the margin of error is the maximum allowed uncertainty in the estimate of a parameter for a design for which the design will still not fail. A similar Google search for "margin of error civil engineering" will provide some samples of margin of error used in civil engineering related discussions although "margin of safety" (to reflect that the margin of error is being used for safety purposes) or the shortened "margin" are also commonly used in civil engineering for this parameter.Fanboy 13:49, Oct 19, 2004 (UTC)

Margin of error is a commonly used statistical term, at least in the UK. Since polling is a very popular use of statistics, it is not surprising that the only place many people have seen it is in polling, but the statistics of polling are the same as those of any other sampling measurement. This page should have a slight re-write to reflect that - it's fine to have most of the stuff as polling examples, but it should show that it is not just a polling topic. Mark Richards 15:33, 19 Oct 2004 (UTC)

It is used virtually as a synonym for confidence interval in all branches of statistics. I made edits earlier but they were changed back by Fadethree - I still think this is not just a polling term. [1] here is an example of a non-polling use. Trollminator 15:45, 19 Oct 2004 (UTC)
Yes, a quick search pulls up many examples non-polling related: general statistics, medical statistics, general statistics, medical salaries data, etc etc. Intrigue 16:00, 19 Oct 2004 (UTC)

I will try to respond to all of the above points here. It is perfectly clear that "margin of error" is a term used well outside the "field" of polling, and I understand that it is often used as a synonym of "confidence interval." I was perhaps overly simplistic at the top of this page, and I didn't want to make my argument there and claim authority by page placement. Let me try to make my point succinct. The issue isn't whether the margin of error is used outside of polling, it is whether it should be. I know that this comment may come off as initially offensive to people who have a grounded and perhaps lifelong attachment to the phrase, but give me a brief chance to make an argument. This argument will not be made from authority, so I don't believe any web page with any number of uses of "margin of error" will necessarily refute it. This argument is for "cutting our losses" with the margin of error at the level of polling and preventing the term from spreading unnecessarily into the vernacular of other disciplines.

I liken "margin of error" to any number of phrases in the language which sound catchy, pique interest, marshall authority, roll off the tongue, and, unfortunately, are inappropriate and misleading. It is no coincidence that an example of these is "statistical dead heat," a bloated and meaningless phrase that stems directly from misinterpretations of the "margin of error." The argument boils down to two points.

  1. We already have established and explicitly defined terms that render the use of the margin of error unnecessary. Confidence intervals and standard errors do the job just fine.
  2. The margin of error, when used as a synonym for "confidence interval," places the illusion of a hard, deterministic border on the interval (look up "margin") which simply does not exist. This encourages mischief and has led to widespread misinformation (see article).

A potential counterargument might be that Wikipedia has a responsibility to report all definitions of the term. I would rephrase to, Wikipedia has a responsibility to report all valid definitions of the term and has an even greater responsibility to address common misconceptions, resolve ambiguities, and establish appropriate use.

I will leave it at this for now, but I hope that the structure of this article and my motivations for reining in the definition are becoming clear. Let me add that this has become enough of an issue to make some note about it in the article, and I would appreciate edits and/or suggestions now or once this discussion tapers off. I also appreciate the civil nature of the discussion thus far. Best, Fadethree 17:57, 19 Oct 2004 (UTC)

Thank you - I, too, appreciate civil debate. I am not sure that I agree that the term is vernacular in other fields - it has a long standing technical definition among professional statisticians outside of polling. I hope that you will agree with the explanation I am adding at the top. Let me know if you do not. Trollminator 18:19, 19 Oct 2004 (UTC)
Thanks, Trollminator. I just took out the reference to confidence intervals because I worry that this will lead to a common misconception in polling, that the margin of error is the confidence interval for every percentage. I'd also like to hold off on other fields until we can list them or integrate "margin of error" into the confidence interval page in a way that makes a clear distinction between uses in polling and uses elsewhere. I think that by noting that this is a term used in polling, it is implicit that it is used elsewhere. I also think that the margin of error in polling is probably the most widely held association. Is this okay? Fadethree 19:08, 19 Oct 2004
I guess I've got 2 more cents to add to this discussion and I may at a later time have time to attempt a proper update to this page if someone else is not able to do so first. On the question of whether margin of error should be used outside of polling, I would have to say yes. The reasons for this are two-fold.
  1. The term Margin of Error is a statistical term that came into use for polling, because of the statistics that polls are based on. Other things that use this term for the same reason should be able to do so.
  2. Other uses of Margin of Error often more appropriately fit the literal definition of the term. Going back to my engineering example from above where margin of error fits the dictionary definition of a margin (From Merian-Webster: 3.a: a spare amount or measure or degree allowed or given for contingencies or special situations) better then polls do. So the margin of error is the spare measure provided in case of errors in calculations/measurments to allow for contingencies and non-ideal situations.
When this page is updated however, I do think that the comments by Mark Richards should be noted as polling is the one use of the term that can be assumed that most people will understand as something that have seen in literature and hence can be used as a strong example in explaining the statistics.Fanboy 5:30, Oct 20, 2004 (UTC)
I look forward to your edits, Fanboy. I still do not think that your points address my first point, that we already have well-established and less misleading terms than "margin of error," but I apparently underestimated the extent to which the term has taken root in other fields. Thank you for your helpful comments, and let's just hope that this article's time on the main page has cleared some of the misconceptions held by some of the laypersons and media folk without oversimplifying the nuanced arguments that we're having here. Best, Fadethree 06:14, 20 Oct 2004 (UTC)

[edit] Bayesian confidence intervals

Margins of error should be compared with Bayesian credible intervals, the interpretation of which is much more natural (ie they give you the probability that a parameter falls within a given range). There doesn't appear to be much material on this in Wikipedia however. I'll write something if I have time.

[edit] Why those formulas aren't in MathML?

 IS THAT A MEDIAWIKI MAJOR ISSUE OF TRULY WORSTE LIMITATION??

We use TeX and html for math here. Maybe someday MathML will be supported, but at this time it isn't, because there's not really a need. There are other issues that are more urgent. —siroχo 11:42, Oct 19, 2004 (UTC)

[edit] Comparing percentages: the probability of leading

For the example where the probability of a candidate leading is computed, it says, "These calculations suggest that the probability that Kerry is "truly" leading is 74.66 percent." Technically, the 74.66% is a (one-sided) p-value, not a probability. That is, it is the probability that we would see a lead of 2% or more for Kerry given that Kerry and Bush are exactly tied. If, for example, we knew it was 47/45 the other way, we could still by random chance draw a sample showing a 2% Kerry lead, even though there is probability 0 he would be actually leading at that moment. Or is this getting too technical for what is supposed to be a relatively nontechnical illustration?

Also, the previous comment "questions" mentions that we don't need to assume p+q=1 for the results, but I think the formula for the standard error of the difference used that assumption. Otherwise, the covariance term would be 2*sqrt(p*q*(1-p)*(1-q))/N since covariance is the product of correlation and the standard deviations.

I took the Bayesian route because I think it is easier for people to understand. The Bayesian vs. Frequentist debates over interpretations is fascinating in the ivory tower but I don't want to risk alienating the lay user by getting into philosophy when it won't make too much of a consequential difference. You are welcome to make an edit, of course. About the p+q=1 bit, you are absolutely right; I was mistaken. Best, Fadethree 19:14, 19 Oct 2004

[edit] Probability of leading

If we're assuming perfect negative correlation, does this not imply that p = 1-q and q=1-p? If so, the formula for the spread considerably simplifies to this:

2\sqrt{\frac{p(1-p)}{N}}

Also, in this limited situation, assuming p is the larger frequency, p-q=p-(1-p)=2p-1, so we can rewrite the above as

\sqrt{\frac{1-(p-q)^2}{N}}.

I can imagine adding a table to the article which shows, for various values of p-q and N, the approximate probability of leading. I will construct such a table, unless someone spots an error in my reasoning. Deco 02:13, 3 Nov 2004 (UTC)

I just realized, since the margin of error is defined in terms of N, we can use this formula to calculate the spread instead, where r is the margin of error and C depends on its confidence:
\frac{r}{C}\sqrt{1-(p-q)^2}
Now we can construct a table of probabilities of leading with margin of error on one axis and difference in percentage points on the other. Since both of these only get up to about 10 in situations where it matters, this table can effectively be complete (maybe have one for 95% confidence margins and one for 99%). This seems like an especially useful table to have. Deco 05:45, 3 Nov 2004 (UTC)
Added those tables. They agree with the text, which is nice. I calculated them using the above formulas, and I believe they're correct. Deco 06:37, 3 Nov 2004 (UTC)
The tables, as stated in the article, apply to two-candidate contests. How applicable are they to multi-candidate races? Canadians, for example, may come to this page from the 39th Canadian General Election page, and our polls will often show five possible responses, at least three of which will have non-negligable shares of the pie. Seems like a good idea to address polls with more than two answers, since that'd cover many situations outside of politics, too. --142.206.2.9 19:28, 19 December 2005 (UTC)

[edit] unclear sentences

"Given the actual percentage difference p − q (2 percent or .02) and the standard error of the difference, above (=.03), use a program like Microsoft Excel to calculate the probability that .02 is greater than zero given a normal distribution with mean 0 and standard deviation .03."

I am trying to wrap my mind around how .02 can be less than zero...
Should this read: "Given the actual percentage difference p − q (2 percent or .02) and the standard error of the difference calculated above (.03), use a program like Microsoft Excel to calculate the probability that a sample from a normal distribution with mean .02 and standard deviation .03 is greater than 0."?
I just changed it to the above because I think it's clearer.
Can someone fix this bit, as I think it's confusing and incomplete. I am not a statistician, although I have a good grounding in maths, but this is where I get lost in the article.
(a) please show an example of the Excel formula used to calculate the probability. Is it NORMDIST? How can the probability (by implication) be independent of the confidence level?
(b) the standard error (deviation?) of the lead is unconnected to the standard error or margin of error in the table. I don't see how using the standard error of the lead can confirm any probability in the table...RodCrosby 19:45, 7 December 2006 (UTC)

"Formally, if the level of confidence is 99 percent, one is 99 percent certain that the "true" percentage in a population is within a margin of error of a poll's reported percentage for a reported percentage of 50 percent. Equivalently, the margin of error is the radius of the 99 percent confidence interval for a reported percentage of 50 percent."

These sentences are quite unclear. I don't even understand what the 50% is about, so I can't fix it yet. --MarSch 18:17, 24 Jun 2005 (UTC)

The wording is a bit odd. What they're saying, for example, is that if I take a poll with a margin of error of 5% and a confidence level of 99%, and 50% of people polled choose A, then we can conclude that in reality there is a 99% chance that between 45% and 55% of all people would choose A. If more or less than 50% of people polled chose A, it's a little bit more complicated - for example, if 99% choose A, then in reality there is a better than 99% chance that it's the actual percentage is between 94% and 100%. It's a little technical - maybe we can gloss over it all somehow without being totally wrong. Deco 01:08, 25 Jun 2005 (UTC)
Aha, I've got it. Fixing. I made it just talk about "at least 99%" probability that it's within one margin of error, which is accurate and hopefully gets the idea across. Deco 01:10, 25 Jun 2005 (UTC)

[edit] Margin of error and "statistical dead heat"

I think the article makes some good points on the misuse of margin of error but perhaps overstates the point. Certainly it's true that the margin of error quoted for the individual percentages is not equal to the margin of error for the "lead"; however, if one knows the margin of error for the lead, this does tell one the confidence one may have that the person leading in the poll is actually leading. If the lead is not different from zero by a certain confidence interval, then one may rationally say that we do not have confidence that the party ostensibly leading is actually leading. Describing this as a "statistical dead heat" is reasonable. The fallacy is to believe that we then have no idea who is leading. If the best estimate for candidate A is leading the best estimate for candidate B then we still conclude that it's more likely than not that A is leading, it's just that we cannot be confident of it if the difference does not fall outside the appropriate confidence interval. The point is that the correct margin of error (for the lead) does tell us something about how seriously to take the reported lead, and the reader of this article may be mislead to think this is not the case.

I agree. As a biologist, I'm always reluctant to drawn any conclusions from results which fall outside the 5% confidence interval. Don't get me wrong, I'm not saying it is not more likely that A is leading B, clearly it is, but in biologist anyway, the generally accepted idea is you just do not drawn conclusions without data supporting at least a 5% confidence interval. The difference is simply not statisticly significant. This is not to say there is NO difference between the two, simply that the data does NOT allow us to draw any conclusions with the degree of confidence necessary for us to pay any heed to the said conclusion. Or put it this was. Our data suggests A is more likely to be leading B but our data does not have enough evidence to suggest this theory is probably correct.

[edit] Title

Shouldn't the title use the word "sampling"? Maurreen 6 July 2005 12:55 (UTC)

I agree that "Margin of sampling error" would be more correct. But the more common usage is "Margin of error", and we should probably follow that in our title. -- Avenue 01:16, 2 May 2006 (UTC)

[edit] Meaning of MOE

I'm mathematically inclined, but not a statistician, and currently I'm a little uncertain of the meaning of MOE. Is the following correct:

Given this information:

Level of confidence: 95 %
Margin of error: 4%
Results:
 A:50%
 B:30%
 C:20%

There is a 95 % chance that A is within 4% of correct.
There is a larger than 95% chance that b is within 4% of correct.
There is an even larger percent chance that C is within 4% of correct.

Is that all correct? Overall this seems to be relatively useless. It seems polls would be better to leave off the MOE, but give the sample size, allowig people who care to calulate the MOE themselves for whatever confidence level they desire. 71.0.197.230 03:07, 24 May 2006 (UTC)

Roughly speaking, that's all correct. And if it was a simple random sample from a large population, you could just give the sample size and let people calculate their own MOEs. But usually most of the the audience for survey results wouldn't know how, or be able to calculate it even if you showed them how. More importantly, many surveys use more complex survey designs, meaning that it becomes much more difficult to calculate MOEs. People would need access to the survey data, details of the sample design, specialised software and statistical expertise to calculate their own margins of error. -- Avenue 04:37, 24 May 2006 (UTC)

[edit] Serious problems with the article

I've been working on and off over the last month to correct various problems with this article. Some of the most glaring problems are now fixed; in particular the complete omission of any reference to the maximum margin of error (when that was what all the figures referred to) or to the assumption of a simple random sample made by all the formulae. Also the limited focus on polls, when margins of error apply to most sample surveys, and the misuse of N instead of n to denote the sample size. There are still some major problems remaining, which I've started to work through: 1) the technically inaccurate descriptions of what a margin of error or confidence interval means; 2) some unstated assumptions about the sample designs assumed by the various formulae and calculations shown; and 3) the fact that most statements in the article apply to only the very few surveys that use simple random sampling with replacement.

Each of these remaining problems on its own would mean the article fails to meet some of the featured article criteria (factual accuracy for problems 1 and 2, and comprehensiveness for problem 3). The third point will be the hardest to address properly, but is probably also the most important. Even if 1 and 2 are fixed, so that our article would not be actively misleading, I believe it would still be of limited use in most practical situations until 3 is dealt with. -- Avenue 15:28, 27 May 2006 (UTC)

It also only has a single reference which appears to be of very limited scope. --MarSch 10:44, 29 May 2006 (UTC)

Another problem: the Newsweek poll was weighted, meaning that the standard formulae (which assume an unweighted simple random sample) shouldn't be applied, but we apply them to this poll anyway. The descriptions of the poll in the external links are very vague and, in particular, they don't say whether a simple random sample was used. -- Avenue 15:06, 31 May 2006 (UTC)

[edit] Running Example issues

I recommend changing the main running example to a 2-person contest. The Bush/Gore/Nader race confuses because it isn't a straightforward enough example. The rest of the discussion never refers again to the Nader statistic--that's a problem and I imagine many readers will wonder about it, making the discussion of the example seeming incomplete. In reality, no presidential race is truly 2-person, but something where the 3rd, 4th etc. candidates are minimal-enough values to be discarded is better. Perhaps with the disclaimer that there is a slight variation from reality.

Nader is a particular problem because of how well-known he is, contra many 3rd party candidates. However, since many situations do involve more than 2 options, a US presidential race of more than 2 that would work well is 1992 when Ross Perot got something like 19%. That works better given how small Nader’s figure was for 2004 and that his figure isn’t high enough to include the whole range of MOE values—his “bottoms out.” Don’t know how US-centric this article is mean to be, but obviously races in other Western nations offer mostly >2 candidate races. Perhaps the 3-person example could be segregated to separate section to keep the 2-person example clearer rather than being intermingled with a 3-person example.

Another issue is that the sections introducing the example do not state the MOE right upfront. That should be immediately stated--anyways--but given that it's an MOE article, all the more. I kept waiting for the shoe to drop, so to speak, on that but it didn't come until (too much) later.

I recommend that the example(s) be spelled out in greater detail to demonstrate that the meaning of MOE as applies to each subject individually, not to the difference between the candidates. I see journalists frequently misstate the MOE calculation as applying to the difference rather than to each candidate. As an example: Candidate A is polled at 53% with B at 47%, MOE of 4. The ranges should be spelled out as for A 57 to 49%; for B 51 to 43%. That gives a concrete example, and can be used to better illustrate the (correct) distinction/importance of considering the application of MOE to candidates A and B individually rather than to the difference of the figure of 4, which would indicate--in that misread of the difference of 6% being (inaccurately reported) outside that MOE of 4.

A brief commentary is made in the dead-heat discussion about the difference and concomitant likelihood, but returning to the running example to spell out the relative (colloquially speaking) the significance (colloquially) of a result for A in the MOE of 57 vs. 49 vs. 53%--how likely is it that 53% is correct vs. 57/49/other percents.

As with many of the math/statistics entries, the issue of very technical explanation vs. man-on-the-street arises. This entry seems to shift from a heuristically useful real-world running example to hardcore calculations. Not sure how to deal with this, but extending the running example to be used in the more technical section about how to calculate the equations could be illustrated better by using the figures from the running examples rather than leaving it more hypothetical and generally abstract--as consistent with the use of a real-world example in the rest of the section. JackWikiSTP (talk) 17:40, 6 May 2008 (UTC)

[edit] Proposed merge with Engineering tolerance

Margin of error is a very different topic from Engineering tolerance. One is the actual amount of random variation in estimates from a sample survey; the other is the allowable variation in physical measurements or properties of engineering components. I can't see any benefit in trying to merge the two articles. Does anyone object to the merge tag being removed? -- Avenue (talk) 14:04, 10 May 2008 (UTC)

Tag now removed. --Salix alba (talk) 17:04, 10 May 2008 (UTC)