The Good Judgment Project

The Good Judgment Project (GJP) is a project "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of Expert Political Judgment: How Good Is It? How Can We Know?), Barbara Mellers (decision scientist and also Tetlock's wife), and Don Moore.[1][2][3] It is a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA) in the United States.[4][5] Predictions are scored using Brier scores.[6] The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."[7]

History

The Good Judgment Project began in July 2011 in collaboration with IARPA-ACE.[8] The first contest began in September 2011.[9][10]

GJP was one of many entrants in the IARPA-ACE tournament, and has repeatedly emerged as the winner in the tournament.[5]

Starting with the summer of 2013, GJP contestants had access to the Integrated Conflict Early Warning System.[6]

People

The co-leaders of the GJP include Philip Tetlock, Barbara Mellers, and Don Moore.[1] The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, and Lyle Ungar.[11] The study employed several thousand people as volunteer forecasters.[12]

Research

A significant amount of research has been conducted based on the Good Judgment Project by the people involved with the project.[13][14] The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row.[12]

Media coverage

GJP has repeatedly been discussed in The Economist.[9][10][15]

GJP has also been covered in the New York Times,[3] the Washington Post,[5][16][17] and Co.Exist.[18]

NPR aired a segment on The Good Judgment Project by the title So You Think You're Smarter Than A CIA Agent, on April 2, 2014.[7]

The Financial Times published an article on the GJP on September 5, 2014.[19]

The Washingtonian published an article that mentioned the GJP on January 8, 2015.[20] The BBC and the Washington Post published articles on the GJP respectively on January 20, 21 and 29, 2015.[21][22] [23]

The Almanac of Menlo Park published a story on the GJP on January 29, 2015.[24]

An article on the GJP appeared on the portal of the Philadelphia Inquirer Philly.com on February 4, 2015.[25]

The book Wiser: Getting Beyond Groupthink to Make Groups Smarter has a section detailing the involvement of the GJP in the tournament run by IARPA.[26]

See also

References

  1. ↑ 1.0 1.1 "Welcome to the Good Judgment ProjectTM". The Good Judgment ProjectTM. Retrieved May 5, 2014.
  2. ↑ "Who’s who in the Good Judgment Project". The Good Judgment ProjectTM. July 27, 2011. Retrieved May 5, 2014.
  3. ↑ 3.0 3.1 Brooks, David (March 21, 2013). "Forecasting Fox". New York Times. Retrieved May 5, 2014.
  4. ↑ "The Project". The Good Judgment ProjectTM. Retrieved May 5, 2014.
  5. ↑ 5.0 5.1 5.2 Horowitz, Michael (November 26, 2013). "Good judgment in forecasting international affairs (and an invitation for season 3)". Washington Post. Retrieved May 5, 2014.
  6. ↑ 6.0 6.1 Dickenson, Matt (November 12, 2013). "Prediction and Good Judgment: Can ICEWS Inform Forecasts?". Predictive Heuristics. Retrieved May 24, 2014.
  7. ↑ 7.0 7.1 Spiegel, Alix. "So You Think You're Smarter Than A CIA Agent". NPR.org. Retrieved 2014-08-18.
  8. ↑ "The idea behind the Good Judgment Project". The Good Judgment ProjectTM. July 27, 2011. Retrieved May 5, 2014.
  9. ↑ 9.0 9.1 "The perils of prediction: Adventures in punditry". The Economist. September 2, 2011. Retrieved May 6, 2014.
  10. ↑ 10.0 10.1 "Monetary policy: How likely is deflation?". The Economist. September 13, 2011. Retrieved May 6, 2014.
  11. ↑ "Team". The Good Judgment ProjectTM. Retrieved May 5, 2014.
  12. ↑ 12.0 12.1 Mellers, Barbara; Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney E. Scott, Don Moore, Pavel Atanasov, Samuel A. Swift, Terry Murray, Eric Stone, Philip E. Tetlock (2014-05-01). "Psychological strategies for winning a geopolitical forecasting tournament". Psychological Science 25 (5): 1106–1115. doi:10.1177/0956797614524255. ISSN 1467-9280. PMID 24659192.
  13. ↑ Ungar, Lyle; Mellers, Barbara; Satopää, Ville; Baron, Jon; Tetlock, Philip E.; Ramos, Jaime; Swift, Sam. "The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions". Association for the Advancement of Artificial Intelligence.
  14. ↑ Ulfelder, Jay (March 27, 2014). "Using the 'Wisdom of (Expert) Crowds' to Forecast Mass Atrocities". Retrieved May 6, 2014.
  15. ↑ "International: Who’s good at forecasts? How to sort the best from the rest". The Economist. November 18, 2013. Retrieved May 6, 2014.
  16. ↑ Ignatius, David (November 1, 2013). "More chatter than needed". Washington Post. Retrieved May 6, 2014.
  17. ↑ Bender, Jeremy (April 3, 2014). "Huge Experiment Finds Regular Folks Predict World Events Better Than CIA Agents". Business Insider. Retrieved May 6, 2014.
  18. ↑ "The Surprising Accuracy Of Crowdsourced Predictions About The Future. Do you know whether Turkey will get a new constitution? It turns out you do: A group of well-informed citizens can predict future events more often than any foreign policy expert or CIA analyst.". Co.exist. April 21, 2014. Retrieved May 6, 2014.
  19. ↑ Harford, Tim (2014-09-05). "How to see into the future". Financial Times. ISSN 0307-1766. Retrieved 2014-09-05.
  20. ↑ Hamilton, Keegan. "How US Agencies Are Using the Web to Pick Our Brains". Washingtonian. Retrieved 2015-01-24.
  21. ↑ Burton, Tara (2015-01-20). "Could you be a ‘super-forecaster’?". BBC Future. Retrieved 2015-01-21.
  22. ↑ Jensen, Nathan (2015-01-21). "Experts see a Republican Senate and fast-track authority for Obama as keys to new trade agreements". The Washington Post. ISSN 0190-8286. Retrieved 2015-01-21.
  23. ↑ Mellers, Barbara; Michael C. Horowitz (2015-01-29). "Does anyone make accurate geopolitical predictions?". The Washington Post. ISSN 0190-8286. Retrieved 2015-01-30.
  24. ↑ "Feature story: Bob Sawyer of Woodside discovers his latent talent in forecasting". Retrieved 2015-03-17.
  25. ↑ Dribben, Melissa; Inquirer Staff Writer (2015-02-04). "Fortune telling: Crowds surpass pundits". Philly.com. Retrieved 2015-02-06.
  26. ↑ Sunstein, Cass R.; Hastie, Reid (2014-12-23). Wiser: Getting Beyond Groupthink to Make Groups Smarter. Harvard Business Review Press. ISBN 9781422122990.

External links