Talk:Exact test

From Wikipedia, the free encyclopedia

This article is within the scope of WikiProject Statistics, which collaborates to improve Wikipedia's coverage of statistics. If you would like to participate, please visit the project page.

[edit] Correctness?

I raise two doubts about this article:

  • Is it really appropriate to restrict "exact test" to nonparametric tests, or is this just some convention of those working in the field which it would be best not to impose on others?
  • Is it appropriate to phrase the test as summing over those outcomes with lowest probabilities, or should it be summing the probabilities of oucomes which are as more extreme than the observed?
Melcombe (talk) 10:34, 13 May 2008 (UTC)

[edit] Can someone check this detail?

I've addes this paragraph:

A simple example of the occasion for this concept may be seen by observing that Pearson's chi-square test is an approximate test. Suppose Pearon's chi-square test is used to to ascertain whether a six-sided die is "fair", i.e. gives each of the six outcomes equally often. If the die is thrown n times, then one "expects" to see each outcome n/6 times. The test statistic is
\sum \frac{(\text{observed}-\text{expected})^2}{\text{expected}} = \sum_{k=1}^6 \frac{(X_k - n/6)^2}{n/6},
where Xk is the number of times outcome k is observed. If the null hypothesis of "fairness" is true, then the probability distribution of the test statistic can be made as close as desired to the chi-square distribution with 5 degrees of freedom by making the sample size n big enough. But if n is small, then the probabilities based on chi-square distributions may not be very close approximations. Finding the exact probability that this test statistic exceeds a certain value then requires combinatorial enumeration of all outcomes of the experiment that result in such a large value of the test statistic. Moreover, it becomes questionable whether the same test statistic ought to be used. A likelihood-ratio test might be preferred as being more powerful, and the test statistic might not be a monotone function of the one above.

Notice that at the end I say "the test statistic might not be a monotone function of the one above". If I were well-versed in this particular problem, I'd know whether it is or is not. If someone knows that, could they add the appropriate information in place of that last sentence? Michael Hardy (talk) 20:06, 16 May 2008 (UTC)

I hope I got this right. Consider two possible outcomes for sample size n = 7:
  • X = (0, 0, 1, 1, 1, 4) – once a 3, a 4 and a 5 are observed each, and twice a 6;
  • X = (0, 0, 0, 2, 2, 3) – twice a 4 and a 5 are observed each, and thrice a 6.
The first has
\chi^2 = \frac{2\times(0-\tfrac{7}{6})^2+3\times(1-\tfrac{7}{6})^2+1\times(4-\tfrac{7}{6})^2}{\tfrac{7}{6}} = \frac{65}{7},
and the second
\chi^2 = \frac{3\times(0-\tfrac{7}{6})^2+2\times(2-\tfrac{7}{6})^2+1\times(3-\tfrac{7}{6})^2}{\tfrac{7}{6}} = \frac{53}{7}.
So the first outcome is the one for which the null hypothesis of fairness is sooner rejected.
Under the null hypothesis of fairness, the first outcome has likelihood C×(1⁄6)7, where C is the number of different ways of getting this X with 7 throws. In the parameter space, the outcome X is most likely if P(k) = Xk/n, giving for the supremum C×(1⁄7)3(4⁄7)4.So the likelihood ratio for the first is:
\Lambda(X) = \frac{\left(\tfrac{1}{6}\right)^7}{\left(\tfrac{1}{7}\right)^3\left(\tfrac{4}{7}\right)^4} = \frac{823543}{71663616}.
Likewise, we find for the second:
\Lambda(X) = \frac{\left(\tfrac{1}{6}\right)^7}{\left(\tfrac{2}{7}\right)^4\left(\tfrac{3}{7}\right)^3} = \frac{823543}{120932352}.
Using this test statistic the second outcome is the one for which the null hypothesis is sooner rejected.
For n up to 6 I haven't found any such "crossovers", and this is one of two I could construct for n = 7.  --Lambiam 22:41, 20 May 2008 (UTC)

Thank you. Just the sort of thing that was needed and that I was too lazy to work out the details of. Now we should think about how to incorporate this information into the article. Michael Hardy (talk) 18:20, 30 May 2008 (UTC)