Post-hoc analysis
From Wikipedia, the free encyclopedia
Post-hoc analysis, in the context of design and analysis of experiments, refers to looking in the data--after the experiment has concluded--for patterns that were not specified a priori. It is also known as data dredging to evoke the sense that the more one looks the more likely something will be found. More subtly, each time a pattern in the data is considered, a statistical test is effectively performed. This greatly inflates the total number of statistical tests and necessitates the use of multiple testing procedures to compensate. However, this is difficult to do precisely and in fact most results of post-hoc analyses are reported as they are with unadjusted p-values. These p-values must be interpreted in light of the fact that they are small and selected subset of a potentially large group of p-values. Results of post-hoc analysis should be explicitly labeled as such in reports and publications to avoid misleading readers.
In practice, post-hoc analysis is usually concerned with finding patterns in subgroups of the sample.
