The Good Judgment Project

The Good Judgment Project (GJP) is a project "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of Expert Political Judgment: How Good Is It? How Can We Know?), Barbara Mellers (decision scientist and also Tetlock's wife), and Don Moore.[1][2][3] It is a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA) in the United States.[4][5] Predictions are scored using Brier scores.[6] The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."[7]

History

The Good Judgment Project began in July 2011 in collaboration with IARPA-ACE.[8] The first contest began in September 2011.[9][10]

GJP was one of many entrants in the IARPA-ACE tournament, and has repeatedly emerged as the winner in the tournament.[5]

Starting with the summer of 2013, GJP contestants had access to the Integrated Conflict Early Warning System.[6]

People

The co-leaders of the GJP include Philip Tetlock, Barbara Mellers, and Don Moore.[1] The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, and Lyle Ungar.[11] The study employed several thousand people as volunteer forecasters.[12]

Research

A significant amount of research has been conducted based on the Good Judgment Project by the people involved with the project.[13][14] The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row.[12]

Media coverage

GJP has repeatedly been discussed in The Economist.[9][10][15]

GJP has also been covered in the New York Times,[3] the Washington Post,[5][16][17] and Co.Exist.[18]

NPR aired a segment on The Good Judgment Project by the title So You Think You're Smarter Than A CIA Agent, on April 2, 2014.[7]

The Financial Times published an article on the GJP on September 5, 2014.[19]

The Washingtonian published an article that mentioned the GJP on January 8, 2015.[20] The BBC and the Washington Post published articles on the GJP respectively on January 20, 21 and 29, 2015.[21][22] [23]

The Almanac of Menlo Park published a story on the GJP on January 29, 2015.[24]

An article on the GJP appeared on the portal of the Philadelphia Inquirer Philly.com on February 4, 2015.[25]

The book Wiser: Getting Beyond Groupthink to Make Groups Smarter has a section detailing the involvement of the GJP in the tournament run by IARPA.[26]

See also

References

  1. 1.0 1.1 "Welcome to the Good Judgment ProjectTM". The Good Judgment ProjectTM. Retrieved May 5, 2014.
  2. "Who’s who in the Good Judgment Project". The Good Judgment ProjectTM. July 27, 2011. Retrieved May 5, 2014.
  3. 3.0 3.1 Brooks, David (March 21, 2013). "Forecasting Fox". New York Times. Retrieved May 5, 2014.
  4. "The Project". The Good Judgment ProjectTM. Retrieved May 5, 2014.
  5. 5.0 5.1 5.2 Horowitz, Michael (November 26, 2013). "Good judgment in forecasting international affairs (and an invitation for season 3)". Washington Post. Retrieved May 5, 2014.
  6. 6.0 6.1 Dickenson, Matt (November 12, 2013). "Prediction and Good Judgment: Can ICEWS Inform Forecasts?". Predictive Heuristics. Retrieved May 24, 2014.
  7. 7.0 7.1 Spiegel, Alix. "So You Think You're Smarter Than A CIA Agent". NPR.org. Retrieved 2014-08-18.
  8. "The idea behind the Good Judgment Project". The Good Judgment ProjectTM. July 27, 2011. Retrieved May 5, 2014.
  9. 9.0 9.1 "The perils of prediction: Adventures in punditry". The Economist. September 2, 2011. Retrieved May 6, 2014.
  10. 10.0 10.1 "Monetary policy: How likely is deflation?". The Economist. September 13, 2011. Retrieved May 6, 2014.
  11. "Team". The Good Judgment ProjectTM. Retrieved May 5, 2014.
  12. 12.0 12.1 Mellers, Barbara; Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney E. Scott, Don Moore, Pavel Atanasov, Samuel A. Swift, Terry Murray, Eric Stone, Philip E. Tetlock (2014-05-01). "Psychological strategies for winning a geopolitical forecasting tournament". Psychological Science 25 (5): 1106–1115. doi:10.1177/0956797614524255. ISSN 1467-9280. PMID 24659192.
  13. Ungar, Lyle; Mellers, Barbara; Satopää, Ville; Baron, Jon; Tetlock, Philip E.; Ramos, Jaime; Swift, Sam. "The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions". Association for the Advancement of Artificial Intelligence.
  14. Ulfelder, Jay (March 27, 2014). "Using the 'Wisdom of (Expert) Crowds' to Forecast Mass Atrocities". Retrieved May 6, 2014.
  15. "International: Who’s good at forecasts? How to sort the best from the rest". The Economist. November 18, 2013. Retrieved May 6, 2014.
  16. Ignatius, David (November 1, 2013). "More chatter than needed". Washington Post. Retrieved May 6, 2014.
  17. Bender, Jeremy (April 3, 2014). "Huge Experiment Finds Regular Folks Predict World Events Better Than CIA Agents". Business Insider. Retrieved May 6, 2014.
  18. "The Surprising Accuracy Of Crowdsourced Predictions About The Future. Do you know whether Turkey will get a new constitution? It turns out you do: A group of well-informed citizens can predict future events more often than any foreign policy expert or CIA analyst.". Co.exist. April 21, 2014. Retrieved May 6, 2014.
  19. Harford, Tim (2014-09-05). "How to see into the future". Financial Times. ISSN 0307-1766. Retrieved 2014-09-05.
  20. Hamilton, Keegan. "How US Agencies Are Using the Web to Pick Our Brains". Washingtonian. Retrieved 2015-01-24.
  21. Burton, Tara (2015-01-20). "Could you be a ‘super-forecaster’?". BBC Future. Retrieved 2015-01-21.
  22. Jensen, Nathan (2015-01-21). "Experts see a Republican Senate and fast-track authority for Obama as keys to new trade agreements". The Washington Post. ISSN 0190-8286. Retrieved 2015-01-21.
  23. Mellers, Barbara; Michael C. Horowitz (2015-01-29). "Does anyone make accurate geopolitical predictions?". The Washington Post. ISSN 0190-8286. Retrieved 2015-01-30.
  24. "Feature story: Bob Sawyer of Woodside discovers his latent talent in forecasting". Retrieved 2015-03-17.
  25. Dribben, Melissa; Inquirer Staff Writer (2015-02-04). "Fortune telling: Crowds surpass pundits". Philly.com. Retrieved 2015-02-06.
  26. Sunstein, Cass R.; Hastie, Reid (2014-12-23). Wiser: Getting Beyond Groupthink to Make Groups Smarter. Harvard Business Review Press. ISBN 9781422122990.

External links