The Good Judgment Project
The Good Judgment Project (GJP) is a project "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of Expert Political Judgment: How Good Is It? How Can We Know?), Barbara Mellers (decision scientist and also Tetlock's wife), and Don Moore.[1][2][3] It is a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA) in the United States.[4][5] Predictions are scored using Brier scores.[6] The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."[7]
History
The Good Judgment Project began in July 2011 in collaboration with IARPA-ACE.[8] The first contest began in September 2011.[9][10]
GJP was one of many entrants in the IARPA-ACE tournament, and has repeatedly emerged as the winner in the tournament.[5]
Starting with the summer of 2013, GJP contestants had access to the Integrated Conflict Early Warning System.[6]
People
The co-leaders of the GJP include Philip Tetlock, Barbara Mellers, and Don Moore.[1] The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, and Lyle Ungar.[11] The study employed several thousand people as volunteer forecasters.[12]
Research
A significant amount of research has been conducted based on the Good Judgment Project by the people involved with the project.[13][14] The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row.[12]
Media coverage
GJP has repeatedly been discussed in The Economist.[9][10][15]
GJP has also been covered in the New York Times,[3] the Washington Post,[5][16][17] and Co.Exist.[18]
NPR aired a segment on The Good Judgment Project by the title So You Think You're Smarter Than A CIA Agent, on April 2, 2014.[7]
The Financial Times published an article on the GJP on September 5, 2014.[19]
The Washingtonian published an article that mentioned the GJP on January 8, 2015.[20] The BBC and the Washington Post published articles on the GJP respectively on January 20, 21 and 29, 2015.[21][22] [23]
The Almanac of Menlo Park published a story on the GJP on January 29, 2015.[24]
An article on the GJP appeared on the portal of the Philadelphia Inquirer Philly.com on February 4, 2015.[25]
The book Wiser: Getting Beyond Groupthink to Make Groups Smarter has a section detailing the involvement of the GJP in the tournament run by IARPA.[26]
See also
- Aggregative Contingent Estimation
- Intelligence Advanced Research Projects Agency
- SciCast
References
- ↑ 1.0 1.1 "Welcome to the Good Judgment ProjectTM". The Good Judgment ProjectTM. Retrieved May 5, 2014.
- ↑ "Who’s who in the Good Judgment Project". The Good Judgment ProjectTM. July 27, 2011. Retrieved May 5, 2014.
- ↑ 3.0 3.1 Brooks, David (March 21, 2013). "Forecasting Fox". New York Times. Retrieved May 5, 2014.
- ↑ "The Project". The Good Judgment ProjectTM. Retrieved May 5, 2014.
- ↑ 5.0 5.1 5.2 Horowitz, Michael (November 26, 2013). "Good judgment in forecasting international affairs (and an invitation for season 3)". Washington Post. Retrieved May 5, 2014.
- ↑ 6.0 6.1 Dickenson, Matt (November 12, 2013). "Prediction and Good Judgment: Can ICEWS Inform Forecasts?". Predictive Heuristics. Retrieved May 24, 2014.
- ↑ 7.0 7.1 Spiegel, Alix. "So You Think You're Smarter Than A CIA Agent". NPR.org. Retrieved 2014-08-18.
- ↑ "The idea behind the Good Judgment Project". The Good Judgment ProjectTM. July 27, 2011. Retrieved May 5, 2014.
- ↑ 9.0 9.1 "The perils of prediction: Adventures in punditry". The Economist. September 2, 2011. Retrieved May 6, 2014.
- ↑ 10.0 10.1 "Monetary policy: How likely is deflation?". The Economist. September 13, 2011. Retrieved May 6, 2014.
- ↑ "Team". The Good Judgment ProjectTM. Retrieved May 5, 2014.
- ↑ 12.0 12.1 Mellers, Barbara; Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney E. Scott, Don Moore, Pavel Atanasov, Samuel A. Swift, Terry Murray, Eric Stone, Philip E. Tetlock (2014-05-01). "Psychological strategies for winning a geopolitical forecasting tournament". Psychological Science 25 (5): 1106–1115. doi:10.1177/0956797614524255. ISSN 1467-9280. PMID 24659192.
- ↑ Ungar, Lyle; Mellers, Barbara; Satopää, Ville; Baron, Jon; Tetlock, Philip E.; Ramos, Jaime; Swift, Sam. "The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions". Association for the Advancement of Artificial Intelligence.
- ↑ Ulfelder, Jay (March 27, 2014). "Using the 'Wisdom of (Expert) Crowds' to Forecast Mass Atrocities". Retrieved May 6, 2014.
- ↑ "International: Who’s good at forecasts? How to sort the best from the rest". The Economist. November 18, 2013. Retrieved May 6, 2014.
- ↑ Ignatius, David (November 1, 2013). "More chatter than needed". Washington Post. Retrieved May 6, 2014.
- ↑ Bender, Jeremy (April 3, 2014). "Huge Experiment Finds Regular Folks Predict World Events Better Than CIA Agents". Business Insider. Retrieved May 6, 2014.
- ↑ "The Surprising Accuracy Of Crowdsourced Predictions About The Future. Do you know whether Turkey will get a new constitution? It turns out you do: A group of well-informed citizens can predict future events more often than any foreign policy expert or CIA analyst.". Co.exist. April 21, 2014. Retrieved May 6, 2014.
- ↑ Harford, Tim (2014-09-05). "How to see into the future". Financial Times. ISSN 0307-1766. Retrieved 2014-09-05.
- ↑ Hamilton, Keegan. "How US Agencies Are Using the Web to Pick Our Brains". Washingtonian. Retrieved 2015-01-24.
- ↑ Burton, Tara (2015-01-20). "Could you be a ‘super-forecaster’?". BBC Future. Retrieved 2015-01-21.
- ↑ Jensen, Nathan (2015-01-21). "Experts see a Republican Senate and fast-track authority for Obama as keys to new trade agreements". The Washington Post. ISSN 0190-8286. Retrieved 2015-01-21.
- ↑ Mellers, Barbara; Michael C. Horowitz (2015-01-29). "Does anyone make accurate geopolitical predictions?". The Washington Post. ISSN 0190-8286. Retrieved 2015-01-30.
- ↑ "Feature story: Bob Sawyer of Woodside discovers his latent talent in forecasting". Retrieved 2015-03-17.
- ↑ Dribben, Melissa; Inquirer Staff Writer (2015-02-04). "Fortune telling: Crowds surpass pundits". Philly.com. Retrieved 2015-02-06.
- ↑ Sunstein, Cass R.; Hastie, Reid (2014-12-23). Wiser: Getting Beyond Groupthink to Make Groups Smarter. Harvard Business Review Press. ISBN 9781422122990.