Ideological Turing Test

The Ideological Turing Test is a concept invented by American economist Bryan Caplan to test whether a political or ideological partisan correctly understands the arguments of his or her intellectual adversaries: the partisan is invited to answer questions or write an essay posing as his opposite number. If neutral judges cannot tell the difference between the partisan's answers and the answers of the opposite number, the candidate is judged to correctly understand the opposing side. The ideological Turing test is so named because of its similarity with the Turing test, a test whereby a machine is required to fool a neutral judge into thinking that it is human.

History

Original formulation

The idea was first mooted by Caplan in 2011^[1] in response to Paul Krugman's claim that, in the context of US politics, liberals understand conservatives (and libertarians) better than conservatives (and libertarians) understand liberals. Borrowing the idea of the Turing test used to judge whether machines can pass themselves off as human, Caplan suggested the ideological Turing test as a way to impartially test Krugman's claim: whichever side understands the other better would perform better on an ideological Turing test. He also offered to take the test himself and offered to bet that libertarians could more easily pass themselves off as liberal than liberals could pass themselves off as libertarian.

Caplan's post was praised by Ilya Somin at the Volokh Conspiracy blog.^[2]

Rise to popularity

Leah Libresco presented the ideological Turing test at Chicago Ideas Week in October 2012.^[3]

Criticism

Noah Smith wrote a blog post critical of ideological Turing tests on the grounds that it rewarded shallow levels of understanding and also rewarded overlooking the contradictions and inconsistencies within an ideology.^[4] Adam Gurri responded to the critique in an article for The Umlaut.^[5]

Attempts at the ideological Turing test

Three versions of the ideological Turing test have been attempted:

1. The identity and ideological affiliation of the author are explicitly known. This may happen if an author who is well known for espousing certain ideological positions writes a piece play-acting as somebody with the opposite ideological position, with a clear indication that this is intended to be an ideological Turing test. A disadvantage of this approach is that readers already know the answer, and hence might be affected by hindsight bias.
2. The identity and ideological affiliation of the author are not known, but it is made clear that this may be an ideological Turing test. Typically, this is done as part of a "contest" where a number of entries are solicited to argue for an ideological position, some from genuine proponents of the position, and some from opponents. People are asked to distinguish genuine proponents from opponents. The proportion of proponents versus opponents may or may not be known in advance. A criticism of this approach was that it made readers overly skeptical compared to what they should be under ordinary circumstances.^[6]
3. The identity and ideological affiliation of the author are not known, and it is not suggested that the piece may be an ideological Turing test. Rather, every attempt is made to deceive readers into believing that they are dealing with a genuine proponent of the position.

Attempts at the ideological Turing test where the identity and ideological affiliation of the author are known in advance

Caplan's blog post inspired a number of tries at the ideological Turing test by prominent bloggers^[7] including those listed below:

Turing test contests: religious and others

Leah Libresco (who was an atheist at the time, but later converted to Catholicism^[11]) set up a contest based on the ideological Turing test idea involving atheists and Christians. Interested contestants were given a set of four questions as prompts and asked to write responses to the questions, both as atheists and as Christians. Fifteen people participated in the contest, about half of them Christian and half atheist.

In the first round, all the fifteen responses written as atheists were published.^[12][13][14] In the second round, all the responses written as Christians were published, albeit not with the same numbering. Readers were asked to fill in a survey specifying their own religious identification and giving their view, for each response whether it was by a Christian or an atheist.

Later, the identities and affiliations of all the participants, as well as how they did in the survey responses, was revealed in a series of blog posts.^[15][16][17]

The ideological Turing test was repeated in 2012 and a resource page recording all iterations of the test was created on the Patheos website.^[18]

In June 2013, Libresco announced that the ideological Turing test would be repeated that year, and the question prompts for the year were based on themes related to Sigmund Freud.^[19]

Attempts where the ideological Turing test nature was deliberately concealed

In July 2013, an anonymous blogger, using the handle "Land Of The Free", published a guest post on Open Borders: The Case (where bloggers generally take pro-open borders positions) claiming to be an opponent of open borders.^[20] The blogger was in fact one of the guest bloggers for the site and a supporter of open borders. However, this fact was known only to the blogger and one site administrator. Other bloggers for the site engaged the anonymous blogger, taking his claims largely at face value. A week later, the blogger revealed his true identity and said that this was an ideological Turing test experiment.^[21]

Historical precedents

Dawes, Singer and Lemon (1972) conducted an experiment along the lines of the ideological Turing test between "hawks" versus "doves" on the Vietnam War in the United States.^[22] They found that each side rejected more statements made by the other side during the experiment than by their own side, on the grounds that the statements were too extreme. However, the degree of exaggeration was not symmetric. Doves significantly exaggerated the extremity of the typical hawk, but hawks showed only a weak tendency to exaggerate the extremity of the typical dove.

References