Exploratory data analysis

From Wikipedia, the free encyclopedia

Exploratory data analysis (EDA) is that part of statistical practice concerned with reviewing, communicating and using data where there is a low level of knowledge about its cause system. It was so named by John Tukey. Many EDA techniques have been adopted into data mining and are being taught to young students as a way to introduce them to statistical thinking.

Tukey's books were notoriously opaque, and so several attempts were made to popularise his EDA ideas. Prominent among these was the Statistics in Society (MDST242) course of The Open University.

Tukey held that too much emphasis in statistics was placed on evaluating and testing given hypotheses (confirmatory data analysis) and that the balance was in need of redressing in favour of using data to suggest hypotheses to test. In particular, confusion of the two types of analysis and employing them on the same set of data can lead to systematic bias owing to the issues endemic in testing hypotheses suggested by the data.

The objectives of EDA are to:

• Suggest hypotheses about the causes of observed phenomena
• Assess assumptions on which statistical inference will be based
• Support the selection of appropriate statistical tools and techniques
• Provide a basis for further data collection through surveys or experiments

The principal graphical tools used in EDA are:

• Box plot
• Histogram
• MultiVari chart
• Run chart
• Pareto chart
• Scatter plot
• Stem-and-leaf plot

The principal quantitative tools are:

• Median polish
• Letter values
• Resistant line
• Resistant smooth
• Rootogram

Contents 1 History 2 Software 3 See also 4 Bibliography 5 References

[edit] History

Many EDA ideas can be traced back to earlier authors, for example:

• Francis Galton - his emphasis on order-statistics and percentiles
• Arthur Bowley - used precursors of the stemplot and five-figure summary (Bowley actually used seven - the median, along with extremes, quartiles and deciles)
• Andrew Ehrenberg's philosophy of Data Reduction (see his book of the same name).

The Open University course Statistics in Society (MDST 242), took the above ideas, and merged them with Gottfried Noether's work, which introduced statistical inference via coin-tossing and the median test.

For details of the above, see John Bibby's book HOTS: History of Teaching Statistics.

[edit] Software

• XLisp-Stat (free software and Lisp based EDA development framework for Mac, PC and X-Windows)
• ViSta (free interactive software based on Xlisp-Stat for EDA)
• DataDesk (free-to-try commercial EDA software for Mac and PC)
• Orange (free component-based software for interactive EDA and machine learning)
• GGobi (free interactive multivariate visualization software linked to R)
• MANET (free Mac-only interactive EDA software)
• Mondrian (free interactive software for EDA)
• Fathom (for high-school and intro college courses)
• TinkerPlots (for upper elementary and middle school students)
• CMU-DAP (Carnegie-Mellon University Data Analysis Package, FORTRAN source for EDA tools with English-style command syntax, 1977)

[edit] See also

• Predictive analytics

[edit] Bibliography

• Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1985). Exploring Data Tables, Trends and Shapes. ISBN 0-471-09776-4.
• Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1983). Understanding Robust and Exploratory Data Analysis. ISBN 0-471-09777-2.
• Tukey, John Wilder (1977). Exploratory Data Analysis. Addison-Wesley. ISBN 0-201-07616-0.
• Velleman, P F & Hoaglin, D C (1981) Applications, Basics and Computing of Exploratory Data Analysis ISBN 0-87150-409-X

[edit] References

• Leinhardt, G., Leinhardt, S., Exploratory Data Analysis: New Tools for the Analysis of Empirical Data, Review of Research in Education, Vol. 8, 1980 (1980), pp. 85-157.