Reliability of Wikipedia

From Wikipedia, the free encyclopedia

The reliability of Wikipedia as an encyclopedia is often assessed in several ways, including statistically, by comparative review, and by analysis of the historical patterns, strengths and weaknesses inherent in the Wikipedia process.

Because Wikipedia is a wiki, and open to collaborative editing by anyone, assessing its reliability requires also examining its ability to detect and rapidly remove false or misleading information. It is also a work in progress[1] that is barely 5 years old as of 2006, with policies, practices and software capabilities also evolving over time.

It should be noted that according to founder Jimmy Wales, encyclopedias as a whole (whether print or online) are not usually appropriate as primary sources and should therefore not be relied upon as authoritative.[1]

At this point in time, a variety of studies to date have tended to suggest that the science entries of Wikipedia are of a similar order of accuracy (and similar rates of both serious and minor errors[2]) to Encyclopædia Britannica, that it provides a good starting point for research, and that articles are in general reasonably sound. However, it does suffer omissions and inaccuracies and sometimes these can be serious.[3] A separate study suggests that in many cases, vandalism is reverted fairly quickly, but that this does not always happen.

Contents

[edit] Areas of reliability

The reliability of Wikipedia covers certain key areas:

  • Accuracy of information provided within articles
  • Comprehensiveness, scope and coverage within articles and in the range of articles
  • Susceptibility to, and exclusion and removal of, false information (a criteria specific to the Wikipedia process)
  • Susceptibility to editorial and systemic bias
  • Identification of reputable third party source references (citations)

Of these, the first (accuracy) and perhaps the second (comprehensiveness) have been the subject of studies, the third (false information durability) was the subject of one recent study; it and the fourth (bias) are strongly disputed on both sides, and the fifth (citations) can be tested within Wikipedia itself.

[edit] The Wikipedia editing model

The Wikipedia model allows anyone to edit, and relies on the large number of well-intentioned editors to overcome issues raised by the smaller number of problematic editors. It is inherent in the Wikipedia model's approach, that poor information can be added, but that over time those editing articles reach strong consensus, and quality improves in a form of group learning, so that substandard edits will very rapidly be removed. This assumption is still being tested and its limitations and reliability are not yet a settled matter – Wikipedia is a pioneer in communal knowledge building of this kind.

This is different to traditional knowledge and publishing, which attempts to limit content creation to a relatively small circle of approved editors in an attempt to exercise strong hierarchical control.

Wikipedia's model of knowledge creation is relatively novel, since widespread collaborative projects of the kind involved were rare until the arrival of the Internet and are still extremely rare on the scale of Wikipedia. Wikipedia itself has developed over time many of the editorial tools found to be useful, based upon trial and error.

It has the potential for extremely rapid growth and harnesses an entire community (much in the same way as other communal projects such as Linux), but goes further in trusting the same community to self-regulate and become better experts at quality control. Wikipedia has harnessed the work of hundreds of thousands of people to produce the world's largest knowledge-based site along with the requisite new software to support this, resulting in approximately 4 million articles written in only 5 years.

For this reason, there has been considerable interest academically and in other areas, from diverse fields such as information technology, business, project management, knowledge acquisition, software programming, other collaborative projects and sociology, to explore whether the Wikipedia model does in fact produce good results, what is revealed about people when they work together this way, and whether the scale of involvement can overcome the obstacles of individual limitations and poor editorship which would otherwise arise.

Another more common set of reasons for inquiry are the growing reliance of many websites which source material from Wikipedia under the GNU Free Documentation Licence (GFDL), and concerns over a major source of information being apparently susceptible to change (including gross misinformation) at whim. The proponents of such concerns tend to be seeking reassurances regarding the quality and reliability of articles, and the degree of usefulness, misinformation or vandalism which should be expected, in order to decide what reliance to place upon it.

[edit] Assessment

[edit] Accuracy of articles

Reliability of information can be assessed by comparison of Wikipedia articles to their parallel articles in other reputable sources.

A common source of reliability criticisms is the open process involved, which means that any article can be modified for better or worse, at any time and no privileged versions of articles currently exist in the main encyclopedia. This fluidity has been assessed by specialists both positively and negatively, as has Wikipedia's model that focuses upon rapid correction rather than initial accuracy.

[edit] Comparative studies

On October 24, 2005, The Guardian published an article "Can you trust Wikipedia?" where a panel of experts were asked to critically review seven entries related to their fields. One article was deemed to have made "every value judgment... wrong", the others receiving marks from 5 to 8 out of a notional ten. (This matches Nature's finding that 4 out of 42, or around 1 in 10 articles in both Wikipedia and Britannica, contain serious errors of understanding.) Of the other six articles reviewed and critiqued, the most common criticisms were:

  1. Poor prose, or ease-of-reading issues (3 mentions)
  2. Omissions or inaccuracies, often small but including key omissions in some articles (3 mentions)
  3. Poor balance, with less important areas being given more attention and vice versa (1 mention)

The most common praises were:

  1. Factually sound and correct, no glaring inaccuracies (4 mentions)
  2. Much useful information, including well selected links, making it possible to "access much information quickly" (3 mentions)

Nature reported in 2005 that science articles in Wikipedia were comparable in accuracy to those in Encyclopædia Britannica. Out of 42 articles, only 4 serious errors were found in Wikipedia, and 3 were found in Encyclopædia Britannica, although more than a hundred lesser errors and omissions were found in each and Wikipedia's articles were often "poorly structured".[3] On March 24, 2006, Britannica provided a rebuttal of this article, labeling it "fatally flawed".[4] (Nature's reply is here (PDF)). However, Kister's Best Encyclopedias 2nd edition (1994) compared the accuracy of Britannica to several other encyclopedias, and concludes that although more accurate than many, it is being ranked lower than encyclopedias such as Encyclopedia Americana, World Book Encyclopedia, and Compton's Encyclopedia.

A web-based survey conducted from December 2005 to May 2006 assessed the "accuracy and completeness of Wikipedia articles". Fifty people (a fairly low response rate) accepted an invitation to assess an article. Of the fifty, thirty-eight (76%) agreed or strongly agreed that the Wikipedia article was accurate, and twenty-three (46%) agreed or strongly agreed that it was complete. Eighteen people compared the article they reviewed to the article on the same topic in the Encyclopædia Britannica. Opinions on accuracy were almost equal between the two encyclopedias (6 favoring Brittanica, 7 favoring Wikipedia, 5 stating they were equal), and eleven (61%) found Wikipedia somewhat or substantially more complete, compared to seven (39%) for Britannica. The survey did not attempt random selection of the participants, and it is not clear how the participants were invited.

The German computing magazine c't performed a comparison of Brockhaus Multimedial, Microsoft Encarta, and Wikipedia in October 2004: Experts evaluated 66 articles in various fields. In overall score, Wikipedia was rated 3.6 out of 5 points ("B-")[5]

In an analysis of online encyclopedias, Indiana University professors Emigh and Herring wrote that "Wikipedia improves on traditional information sources, especially for the content areas in which it is strong, such as technology and current events."[6].

[edit] Subjective expert opinion

[edit] Librarian views

A 2006 review of Wikipedia by LibraryJournal.com, using a panel of librarians, "the toughest critics of reference materials, whatever their format",[7] asked "long standing reviewers" to evaluate three areas of Wikipedia (popular culture, current affairs, and science), and concluded: "While there are still reasons to proceed with caution when using a resource that takes pride in limited professional management, many encouraging signs suggest that (at least for now) Wikipedia may be granted the librarian’s seal of approval". A reviewer who "decided to explore controversial historical and current events, hoping to find glaring abuses" concluded "I was pleased by Wikipedia’s objective presentation of controversial subjects" but that "as with much information floating around in cyberspace, a healthy degree of skepticism and skill at winnowing fact from opinion are required." Other reviewers noted that there is "much variation" but "good content abounds".

The library at Trent University, Ontario, Canada states of Wikipedia that many articles are "long and comprehensive", but that there is "a lot of room for misinformation and bias [and] a lot of variability in both the quality and depth of articles". It adds that Wikipedia has advantages and limitations, that it has "excellent coverage of technical topics" and articles are "often added quickly and, as a result, coverage of current events is quite good", comparing this to traditional sources which are unable to achieve this task. It concludes that depending upon the need, one should think critically and assess the appropriateness of ones sources, "whether you are looking for fact or opinion, how in-depth you want to be as you explore a topic, the importance of reliability and accuracy, and the importance of timely or recent information", and adds that Wikipedia can be used in any event as a "starting point".

An article for the Canadian Library Association (CLA) [8] discusses the Wikipedia approach, process and outcome in depth, commenting for example that in controversial topics, "what is most remarkable is that the two sides actually engaged each other and negotiated a version of the article that both can more or less live with". The author comments that:

"in fact Wikipedia has more institutional structure than at first appears. Some 800 experienced users are designated as administrators, with special powers of binding and loosing: they can protect and unprotect, delete and undelete and revert articles, and block and unblock users. They are expected to use their powers in a neutral way, forming and implementing the consensus of the community. The effect of their intervention shows in the discussion pages of most contentious articles. Wikipedia has survived this long because it is easier to reverse vandalism than it is to commit it..."

Information Today (March 2006) cites librarian Nancy O’Neill (principal librarian for Reference Services at the Santa Monica Public Library System) as saying that "there is a good deal of skepticism about Wikipedia in the library community" but that "she also admits cheerfully that Wikipedia makes a good starting place for a search. You get terminology, names, and a feel for the subject." [9]

In a 2004 interview with The Guardian, self-described information specialist and internet consultant[10] Philip Bradley said that he would not use Wikipedia and is "not aware of a single librarian who would. The main problem is the lack of authority. With printed publications, the publishers have to ensure that their data are reliable, as their livelihood depends on it. But with something like this, all that goes out the window."[11]

[edit] Academia

Academic circles have not been exclusively dismissive of Wikipedia as a reference. Wikipedia articles have been referenced in "enhanced perspectives" provided on-line in Science. The first of these perspectives to provide a hyperlink to Wikipedia was "A White Collar Protein Senses Blue Light",[12] and dozens of enhanced perspectives have provided such links since then. However, these links are offered as background sources for the reader, not as sources used by the writer, and the "enhanced perspectives" are not intended to serve as reference material themselves.

The Gould Library at Carleton College in Minnesota has a page describing the use of Wikipedia in academia. It asserts that "Wikipedia is without question a valuable and informative resource", but that "there is an inherent lack of reliability and stability" to its articles, again drawing attention to similar advantages and limitations as other sources. As with other reviews it comments that one should assess ones sources and what is desired from them, and that "Wikipedia may be an appropriate resource for some assignments, but not for others". It cited Jimmy Wales' view that Wikipedia may not be an ideal as a source for all academic uses, and (as with other sources) suggests that at the least, one strength of Wikipedia is that it provides a good starting point for current information on a very wide range of topics.

[edit] Editors of other encyclopedias

In a 2004 piece called "The Faith-Based Encyclopedia," former Britannica editor Robert McHenry criticized the wiki approach, writing:

"[H]owever closely a Wikipedia article may at some point in its life attain to reliability, it is forever open to the uninformed or semiliterate meddler… The user who visits Wikipedia to learn about some subject, to confirm some matter of fact, is rather in the position of a visitor to a public restroom. It may be obviously dirty, so that he knows to exercise great care, or it may seem fairly clean, so that he may be lulled into a false sense of security. What he certainly does not know is who has used the facilities before him." [13]

Similarly, Encyclopædia Britannica's executive editor, Ted Pappas, was quoted in The Guardian as saying: "The premise of Wikipedia is that continuous improvement will lead to perfection. That premise is completely unproven."[11]

[edit] Other

Information Today (March 2006) comments[9] on Wikipedia and Britannica that it is comparing "Apples and Oranges" and that:

"[E]ven the revered Encyclopaedia Britannica is riddled with errors, not to mention the subtle yet pervasive biases of individual subjectivity and corporate correctness... There is no one perfect way. Britannica seems to claim that there is. Wikipedia acknowledges there’s no such thing. Librarians and information professionals have always known this. That’s why we always consult multiple sources and counsel our users to do the same."

BBC technology specialist Bill Thompson wrote[14] that "Most Wikipedia entries are written and submitted in good faith, and we should not let the contentious areas such as politics, religion or biography shape our view of the project as a whole", that it forms a good starting point for serious research but that:

"No information source is guaranteed to be accurate, and we should not place complete faith in something which can so easily be undermined through malice or ignorance... That does not devalue the project entirely, it just means that we should be sceptical about Wikipedia entries as a primary source of information... It is the same with search engine results. Just because something comes up in the top 10 on MSN Search or Google does not automatically give it credibility or vouch for its accuracy or importance."

He adds the observation that since most popular online sources are inherently unreliable in this way, one byproduct of the information age is a wiser audience who are learning to check information rather than take it on faith due to its source, leading to "a better sense of how to evaluate information sources".

[edit] Removal of false information

Perhaps the most notorious test of false information was the John Seigenthaler Sr. controversy in 2005, when a biography of a famous writer and journalist was found to contain libellous hoax material that had gone undetected for over 4 months.

In an informal media test of Wikipedia's ability to detect misinformation, an anonymous blogger tested Wikipedia by inserting subtly erroneous facts into obscure articles, and stated that its process "isn't really a fact-checking mechanism so much as a voting mechanism", and that material which did not appear "blatantly false" may be accepted as true.[15] Wikipedians by and large responded with anger at what was considered by many to be an unfair trial which had deliberately focused on obscure less reviewed articles; the blogger responded that the test was fair.

Viégas, Wattenberg, and Dave (2004) studied the flow of editing in the Wikipedia model, with emphasis on breaks in flow (from vandalism or substantial rewrites), showing the dynamic flow of material over time. They found that most acts of vandalism during May 2003 were repaired within minutes. However it is unclear whether or not this finding applies to all forms of vandalism, including so-called 'sneaky' vandalism (which appears like genuine information and is by nature harder to detect). Lih (2004) compared articles before and after press mention, and found that external referencing of articles incentivized editors to higher quality work.

(Also see this 2002 study by IBM which found that most vandalism on the English Wikipedia is reverted within five minutes. Official results)

[edit] Coverage

Wikipedia has been accused of deficiencies in comprehensiveness because of its voluntary nature, and of reflecting the systemic biases of its contributors. Encyclopædia Britannica editor-in-chief Dale Hoiberg has argued that "people write of things they're interested in, and so many subjects don't get covered; and news events get covered in great detail. The entry on Hurricane Frances was five times the length of that on Chinese art, and the entry on Coronation Street was twice as long as the article on Tony Blair."[11] (As of December 2005, this is no longer the case.) Former Nupedia editor-in-chief Larry Sanger stated in 2004, "when it comes to relatively specialized topics (outside of the interests of most of the contributors), the project's credibility is very uneven."[16]

Wikipedia has been praised for making it possible for articles to be updated or created in response to current events. For example, the then-new article on the 2004 Indian Ocean earthquake on its English edition was cited often by the press shortly after the incident.[citation needed] Its editors have also argued that, as a website, Wikipedia is able to include articles on a greater number of subjects than print encyclopedias may.[17]

[edit] Broad assessments

Seeing Wikipedia as developing its own methodology, several commentators have drawn a middle ground, that it contains much valuable knowledge and has some reliability, even if the degree is not yet assessed with certainty.

As well as many of the librarian and academic reviewers of Wikipedia cited above, people taking such a view include Danah Boyd, who in 2005 discussed Wikipedia as an academic source, concluding that "[i]t will never be an encyclopedia, but it will contain extensive knowledge that is quite valuable for different purposes",[18] and Bill Thompson who stated "I use the Wikipedia a lot. It is a good starting point for serious research, but I would never accept something that I read there without checking." [14]

Information Today's March 2006 article[9] concludes on a similar theme:

"The inconvenient reality is that people and their products are messy, whether produced in a top-down or bottom-up manner. Almost every source includes errors... Many non-fiction books are produced via an appallingly sloppy process... In this author’s opinion, the flap over Wikipedia was significantly overblown, but contained a silver lining: People are becoming more aware of the perils of accepting information at face value. They have learned not to consult just one source."

Dan Gillmor, a Silicon Valley commentator and author commented in October 2004 that, "I don't think anyone is saying Wikipedia is an absolute replacement for a traditional encyclopedia. But in the topics I know something about, I've found Wikipedia to be as accurate as any other source I've found." [11]

Referencing Linus' law of open-source development, Larry Sanger stated on Kuro5hin in 2001 that "Given enough eyeballs, all errors are shallow."[19]

Likewise, technology figure Joi Ito wrote on Wikipedia's authority, "[a]lthough it depends a bit on the field, the question is whether something is more likely to be true coming from a source whose resume sounds authoritative, or a source that has been viewed by hundreds of thousands of people (with the ability to comment) and has survived."[20]

[edit] References

  1. ^ a b Wikipedia: "A Work in Progress" December 14, 2005
  2. ^ One of the studies, by Nature, identified that Wikipedia and Encyclopædia Britannica had a comparable level both of serious errors (4 and 4 respectively in 42 articles) and also of lesser errors and omissions (162 and 123 respectively) in its science entries.
  3. ^ a b "Wikipedia survives research test", BBC News, BBC, December 15, 2005.
  4. ^ "Journal Nature study "fatally flawed" says Britannica", WikiNews, Wikipedia Foundation, March 24, 2006.
  5. ^ Michael Kurzidim: Wissenswettstreit. Die kostenlose Wikipedia tritt gegen die Marktführer Encarta und Brockhaus an, in: c't 21/2004, October 4, 2004, S. 132-139.
  6. ^ William Emigh and Susan C. Herring, "Collaborative Authoring on the Web: A Genre Analysis of Online Encyclopedias", paper presented at the 39th Hawaii International Conference on System Sciences, 2004.
  7. ^ I want my Wikipedia! Library Journal April 2006
  8. ^ Peter Binkley, “Wikipedia Grows Up”, Feliciter 52 (2006), no. 2, 59-61 [1]
  9. ^ a b c
  10. ^ Self description taken from blog biography, [2]
  11. ^ a b c d Simon Waldman, "Who knows?", The Guardian, October 26, 2004.
  12. ^ Linden, Hartmut (2002-08-02). A White Collar Protein Senses Blue Light. Science. Retrieved on 2005. (subscription access only)
  13. ^ Robert McHenry, "The Faith-Based Encyclopedia", Tech Central Station, November 15, 2004.
  14. ^ a b What is it with Wikipedia? 16 December 2005.
  15. ^ Anonymous blogger, "How Authoritative is Wikipedia", Dispatches from the Frozen North, September 4, 2004. Edits for this test can be found [3].
  16. ^ Larry Sanger, "Why Wikipedia Must Jettison Its Anti-Elitism", Kuro5hin, December 31, 2004.
  17. ^ "Wikipedia:Replies to common objections", Wikipedia, 22:53 April 13, 2005.
  18. ^ Danah Boyd, "Academia and Wikipedia", Many-to-Many, January 4, 2005.
  19. ^ "Wikipedia is wide open. Why is it growing so fast? Why isn't it full of nonsense?", September 24, 2001.
  20. ^ Joi Ito, "Wikipedia attacked by ignorant reporter", Joi Ito's Web, August 29, 2004.

[edit] See also

Wikipedia-specific articles
General articles
Retrieved from "http://en.wikipedia.org../../../r/e/l/Reliability_of_Wikipedia_e12d.html"