Wikipedia talk:WikiProject Medicine/Reliable sources

From Wikipedia, the free encyclopedia

Contents

[edit] Intent?

I'm curious as to the intent of this page; do you seek to create an exhaustive list of magazines and books to use as resources? Or is this page a group of examples, in which case I wonder why it is needed in addition to the normal pages like WP:RS? (Radiant) 13:39, 14 November 2006 (UTC)

This page started as a place to put the "reliable sources" portion of the WP:MEDMOS, which I felt were not style issues and should go elsewhere. It is not intended to have exhaustive list and certainy not for all books/journals as there are simply too many that are of excellent quality. The only reason to list a handful of top journals is to highight those that are regarded as the cream.
Recent discussions on talk WP Medicine have asked:
  • Are primary sources better than secondary sources? In particular, editors who cite papers covering basic science possibly being interpreted for its affects in the human population.
  • Is XYZ a reliable web site that we should consider either using as a reference or providing external links?
This page could be used to capture consensus on those issues. I could also serve as a resource of sites useful to WP Medicine editors. It is currently very much a work-in-progress. I hoped that others might contribute. Perhaps the links to it are too subtle at present. Colin°Talk 14:17, 14 November 2006 (UTC)
  • I'm no expert on the matter of medicine, but if you want consensus on this page you're definitely going to need more input. While you're technically correct about source matters not belonging in the MOS, if that page is more active than this one it's not a big problem - it is common on Wikipedia for pages to have a somewhat broader scope than their title suggests. (Radiant) 15:10, 15 November 2006 (UTC)

[edit] Previous version at RS

WP:RS was substantially rewritten on 1st December 2006 (diff). The following text, which had existed for a while and which we had borrowed for these guidelines, was lost:

[edit] Physical sciences, mathematics and medicine

[edit] Cite peer-reviewed scientific publications and check community consensus

Scientific journals are the best place to find primary source articles about experiments, including medical studies. Any serious scientific journal is peer-reviewed. Many articles are excluded from peer-reviewed journals because they report what is in the opinion of the editors unimportant or questionable research. In particular be careful of material in a journal that is not peer-reviewed reporting material in a different field. (See the Marty Rimm and Sokal affairs.)

The fact that a statement is published in a refereed journal does not make it true. Even a well-designed experiment or study can produce flawed results or fall victim to deliberate fraud. (See the Retracted article on neurotoxicity of ecstasy and the Schön affair.)

Honesty and the policies of neutrality and No original research demand that we present the prevailing "scientific consensus". Polling a group of experts in the field wouldn't be practical for many editors but fortunately there is an easier way. The scientific consensus can be found in recent, authoritative review articles or textbooks and some forms of monographs.

There is sometimes no single prevailing view because the available evidence does not yet point to a single answer. Because Wikipedia not only aims to be accurate, but also useful, it tries to explain the theories and empirical justification for each school of thought, with reference to published sources. Editors must not, however, create arguments themselves in favor of, or against, any particular theory or position. See Wikipedia:No original research, which is policy. Although significant-minority views are welcome in Wikipedia, the views of tiny minorities need not be reported. (See Wikipedia:Neutral Point of View.)

Make readers aware of any uncertainty or controversy. A well-referenced article will point to specific journal articles or specific theories proposed by specific researchers.

[edit] In science, avoid citing the popular press

The popular press generally does not cover science well. Articles in newspapers and popular magazines generally lack the context to judge experimental results. They tend to overemphasize the certainty of any result, for instance presenting a new experimental medicine as the "discovery of the cure" of a disease. Also, newspapers and magazines frequently publish articles about scientific results before those results have been peer-reviewed or reproduced by other experimenters. They also tend not to report adequately on the methodology of scientific work, or the degree of experimental error. Thus, popular newspaper and magazine sources are generally not reliable sources for science and medicine articles.

What can a popular-press article on scientific research provide? Often, the most useful thing is the name of the head researcher involved in a project, and the name of his or her institution. For instance, a newspaper article quoting Joe Smith of the Woods Hole Oceanographic Institution regarding whales' response to sonar gives you a strong suggestion of where to go to find more: look up his work on the subject. Rather than citing the newspaper article, cite his published papers.

[edit] Which science journals are reputable?

One method to determine which journals are held in high esteem by scientists is to look at impact factor ratings, which track how many times a given journal is cited by articles in other publications. Be aware, however, that these impact factors are not necessarily valid for all academic fields and specialties.

In general, journals published by prominent scientific societies are of better quality than those produced by commercial publishers. The American Association for the Advancement of Science's journal Science is among the most highly regarded; the journals Nature and Cell are notable non-society publications.

Keep in mind that even a reputable journal may occasionally post a retraction of an experimental result. Articles may be selected on the grounds that they are interesting or highly promising, not merely because they seem reliable.

[edit] arXiv preprints and conference abstracts

There are a growing number of sources on the web that publish preprints of articles and conference abstracts, the most popular of these being arXiv. Such websites exercise no editorial control over papers published there. For this reason, arXiv (or similar) preprints and conference abstracts should be considered to be self-published, as they have not been published by a third-party source, and should be treated in the same way as other self-published material. See the section above on self-published sources. Most of them are also primary sources, to be treated with the caution as described in various sections of this guideline.

Researchers may publish on arXiv for different reasons: to establish priority in a competitive field, to make available newly developed methods to the scientific community while the publication is undergoing peer-review (a specially lengthy process in mathematics), and sometimes to publish a paper that has been rejected from several journals or to bypass peer-review for publications of dubious quality. Editors should be aware that preprints in such collections, like those in the arXiv collection, may or may not be accepted by the journal for which they were written — in some cases they are written solely for the arXiv and are never submitted for publication. Similarly, material presented at a conference may not merit publication in a scientific journal.

[edit] Evaluating experiments and studies

There are techniques that scientists use to prevent common errors, and to help others replicate results. Some characteristics to look for are experimental control (such as placebo controls), and double-blind methods for medical studies. Detail about the design and implementation of the experiment should be available, as well as raw data. Reliable studies don't just present conclusions.

[edit] Popular science and medicine books

Responding to "How do you know what the point is? Discuss this in Talk before deleting."

I wish to remove the line that says "Some well known and respected popular science authors include Richard Dawkins and Stephen Jay Gould." Can the author of this sentence please explain the point of including just a couple of names out of potentially hundreds of worthy authors? Colin°Talk 14:27, 13 April 2007 (UTC)

Well I originally gave three names in this edit but Sandy removed Oliver Sacks. Colin°Talk 14:27, 13 April 2007 (UTC)
I don't care much for Richard Dawkins myself and am now questioning the value of providing a short list. This page needs general guidelines instead Colin°Talk 14:27, 13 April 2007 (UTC)
Yeah. I see what you mean. Just remove it. Colin°Talk 14:27, 13 April 2007 (UTC)
Done. Colin°Talk 14:27, 13 April 2007 (UTC)

(Apologies for the sarcasm – just a bit of fun).

Nice debate, sound conclusion :-) As an example, Sacks is infamous for his non-standard and sensationalist views on TS (I've heard other physicians opining on his writing). While we're on the topic: I don't understand either of these edits. [1] We need to get back to, Avoid citing the popular press, as they usually get it wrong. [2] The Merck Manual is an utter and total inaccurate wreck when it comes to TS; I hope it's better in other areas. SandyGeorgia (Talk) 14:33, 13 April 2007 (UTC)

The reason for including a few names is that the text gives the reader no way to distinguish between reliable books and unreliable books, or primary sources and tertiary sources. Giving a few examples gives them an idea of what the text is referring to. The reason for choosing 2 or 3 examples out of potentially hundreds of worthy authors is that you have to stop somewhere. Nbauman 15:17, 13 April 2007 (UTC)
OK, maybe we can approach that from a different angle, then. Instead of giving examples of book authors who may be reliable (and I certainly disagree on using Sacks except in very limited contexts, as I did on TS)—what if we instead give an example of a popular or vanity press book that shouldn't be used ? There are legions, for example, in the realm of ADHD. Patty Duke's book is often used to cite bipolar articles—is that reasonable? In other words, maybe we can say what we "shouldn't" use instead of what we should ??? SandyGeorgia (Talk) 15:22, 13 April 2007 (UTC)
That path may lead to conflict. Nbauman is right that it would be helpful to clarify primary from secondary and tertiary. I recently read a good book on Aspirin (ISBN 0747570833) that is thoroughly sourced. It certainly isn't a primary source but is a bit of a mix between secondary and tertiary. It is probably a reasonable source for historical material on the aspirin page but since the author isn't a scientist or doctor, would be a poor choice for the pharmacological material. Very few books are primary sources. Those that are tend to be quoted as a source of original ideas, which is why Sacks and Dawkins were on the list, though the latter hasn't much to say that would concern this wiki project. Colin°Talk 15:38, 13 April 2007 (UTC)
Not sure how to fix this, so will offer an example (not one I'd want to highlight, though). Comings, Tourette Syndrome and Human Behavior. He owns the vanity press, Hope Press. He couldn't get his work accepted by peers, so he self-published. (For a scathing review of his methodological flaws, see this article.) He's a recognized physician in the field (more infamous than famous),so someone could argue that he's a reliable source. I suppose he would be covered under WP:RS self-published sources, but that's the kind of example I'm concerned about. In terms of wacky theories, anyone can get a book published, so maybe we can find an example that's not self-published as Comings is. And, the problem with Sacks is his focus on sensationalized aspects and his personal views. SandyGeorgia (Talk) 17:01, 13 April 2007 (UTC)
Scientists aren't always the best people for describing the scientific literature and qualities like reliability. The people with the expertise to do that are science librarians. Science librarians work with scientists, and many science librarians have had experience as bench scientists and then went on to be trained as librarians. There are many published reference books on library science that cover the subject of this entry much better than this entry does, and have solved many of the problems that these editors are struggling with. I wish some medical librarians would work on this entry. Nbauman 15:17, 13 April 2007 (UTC)
I don't think we have a medical librarian on board, but I do know someone who may be able to look at your concerns. SandyGeorgia (Talk) 15:22, 13 April 2007 (UTC)
I read the book reviews in Science, New England Journal of Medicine, and BMJ. They often review popular books and recommend them strongly. For example, the NEJM last month reviewed "Skin: A Natural History" and a book about Jack Kevorkian. BMJ has been particularly supportive of books that give the patient's perspective. If the reviewers of the major journals, who are academic experts in the field, recommend a popular book, who are we to tell people not to read them? Nbauman

[edit] Essay or guideline?

This page is currently a bit too much like an essay than a guideline. That might be fine if we want to keep it an essay, and we may. If not, then the text needs to be condensed and keep to the point. There's a lot more that can be said on this topic and I really welcome other contributors. It might be best to let the text expand a bit before we start refining it back down. That said, if something important got deleted or watered down, then we should bring it back. Colin°Talk 14:44, 13 April 2007 (UTC)

[edit] Newspapers

I deleted the claim that broadsheets can be reliable sources of medical information, while tabloids are not. Some tabloids are excellent sources of medical information, and conversely.

Here's a good example from today's New York Daily News, about the automobile accident of New Jersey Governor Jon Corzine, who was not wearing a seat belt.

[3] A difficult recovery that could take months, by Christina Boyle, New York Daily News, April 14th 2007.

This is what teachers call a "teachable moment," an opportunity to teach an important message because the subject has everyone's attention. This story explains exactly how someone is hurt in an auto accident if they're not wearing a seat belt, and it explains Corzine's injuries in meaningful detail. I've read hundreds of accident reports in the engineering literature, and this tabloid news story covers all the essential points. Nbauman 14:12, 14 April 2007 (UTC)

Looks reasonably well reported - but that is just the issue. All the journalist did was parrot what she was told by the doctor, while probably simplifying the language and possibly making mistakes along the way. I'm not sure how such a story could form the basis of a source on a medical article - just on a bio of the poor man himself. I appreciate that this is just an example where you considered the reporting to be of a good quality.
Perhaps US tabloids are different to UK ones. Some quality broadsheets employ journalists that write original material that may be worth citing on Wikipedia. Though, like the article above, I can't think of a circumstance where I'd do so on a medical article other than in a history/bio section. The journalist Ben Goldacre has a column Bad Science that frequently points out the shortcomings in the UK newspapers, broadsheet and tabloid.
So while I'd support your deletion of broadsheets, I wouldn't support the claim that "Some tabloids are excellent sources of medical information". Not from my experience, anyway. Colin°Talk 15:01, 14 April 2007 (UTC)
I have to agree. Reporting on a current event is different than sourcing a medical article. A report like that might be useful somewhere in an article about seat belt safety, but shouldn't be the sole basis for medical statements. SandyGeorgia (Talk) 15:18, 14 April 2007 (UTC)
You say, "All the journalist did was parrot what she was told by the doctor, while probably simplifying the language," as if that were a trivial task.
That's not all she did. First, she had to identify one of the important scientific issues, which in this case was, "How do people get injured in automobile accidents, and how do seat belts prevent those injuries?"
Second, she had to find a doctor who was authoritative and knowledgeable, just as a journal editor would have to find an authoritative author for a review article. She didn't get any old doctor, she interviewed an academic doctor at a major critical care center.
Third, she had to get the doctor's intended meaning and quotes right. That's not easy. If she didn't do that right, the doctor would write a letter of correction. I've read a lot of the peer-reviewed auto safety engineering literature, and in my reading of this article, she seems to have gotten it right.
You say, "and possibly making mistakes along the way." You don't know that. That's speculation based on no evidence. The question at issue is whether she made mistakes. In my reading, she didn't.
The reason this news story is so important is that it performs a function of public health education. Most of the medical professional societies in the specialties that deal with automobile trauma have argued for educating the public about the dangers of automobile accidents and the protective effects of seat belts. This article is directly fulfilling that purpose. She quoted a doctor saying so directly:
Had Corzine been belted in, his injuries would have been much less severe. "A seat belt keeps your body fixed to the seat so you wouldn't move forward so much," Shou said. "A seat belt can significantly decrease the chances of these type of injuries."
This is exactly the kind of statements doctors say they want newspapers to print.
But go back to the original question, which is, are tabloid newspapers accurate and reliable? This example from the New York Daily News is clearly accurate and reliable, like all the other medical news I've read in the Daily News. According to the BMJ, and my own sampling, the U.K. broadsheets are not always that accurate. Neither is the New York Times. So any generalized conclusion about the accuracy of tabloids vs. broadside is contrary to fact. Nbauman 17:04, 14 April 2007 (UTC)
I don't think we disagree that much. We just have different experiences. It is great that your New York Daily News is so good. Our tabloids are truly awful. I agree many of the broadsheets are awful too - even the journalists that are supposed to be the house science or medicine reporter. Colin°Talk 17:30, 14 April 2007 (UTC)
So, let's find a way to make our statements as generally correct as possible; this "tabloid" may have gotten it right, but unfortunately, most don't. There are situations in articles where quoting the popular press is fine, but to the extent possible, peer reviewed literature would be preferred. In my experience, *most* of the time, the popular press gets it wrong in subtle or blatant ways. Can we find some wording around the notion that there are acceptable uses, but peer-reviewed medical literature should be consulted ? SandyGeorgia (Talk) 18:00, 14 April 2007 (UTC)

I've removed the line:

Even peer-reviewed journals like the New England Journal of Medicine cite articles in newspapers like the New York Times and Wall Street Journal.

This may be true but I've yet to see an example. If they do, surely it is more for certain social, historical or biographical information rather than for medical facts? Some examples would help. I've also removed the line:

Newspapers should be judged on the facts, not on prejudices.

Which IMO is not written in a suitable tone.

Finally, I've added a line to clarify where I think there is consensus for using and not using newspapers. Colin°Talk 11:37, 16 April 2007 (UTC)

Colin wrote: This may be true but I've yet to see an example.
See the example that I cited below of the NYT article on Guidant being quoted in the NEJM article on Guidant. Nbauman 13:27, 16 April 2007 (UTC)
Yes but that's current-affairs. It's a story about a certain individual and the citation backs up the story. And, as you note later, the "Perspective" section of NEJM might not be peer reviewed (and hence the term "peer reviewed journals" is a simplification). The current text, that you've restored, gives the impression that medical journals routinely cite newspapers. They may very occasionally cite them for news but not for medical facts. Please reconsider your restoration of that text. I'm not going to edit-war over this. Can anyone else help us reach consensus? Colin°Talk 13:54, 16 April 2007 (UTC)
I can't work on it today (or maybe even tomorrow) because I'm flooded from the Nor'easter, but one way to address Nbauman's concerns may be to go back to the top of the article and make it all more about what the highest-quality and medical sources do right, and less about what the others may do wrong (taking care not to overgeneralize). Then we can revisit the newspapers and other sections in that context. If we approach it that way, we may also reduce some redundancy. For example, "The quality of newspaper coverage of medicine ranges from excellent to irresponsible, and they should be verified like any other sources" can be said of most of the other sources (not just newspapers), so we should rephrase in a way that we're not just repeating WP:RS and/or singling out newspapers, rather emphasizing what are the best sources and why. I hope this makes sense; I haven't had much sleep :-) SandyGeorgia (Talk) 15:48, 16 April 2007 (UTC)
I'm sure this guideline will be rewritten more than once before we get it right. There are so many poor sources, that I can't see how the guideline would be useful without mentioning them. After all, people are going to quote this guideline when disputing a source, not when congratulating someone for using a top quality one. One thing that might help is to clarify that this guideline is for the medical facts in medical articles. The social, biographical, current-affairs, etc information can come from sources that meet the general WP:RS. Colin°Talk 17:11, 16 April 2007 (UTC)

Nbauman, earlier you said "who are we to tell people not to read them". I think this may be one source of our disagreement. This is not an article to advise people what to read (whether for pleasure or for research for a WP article). It is solely concerned with what we should cite. Certain books, newspapers, magazines and blogs may be a reliable source of medical information. They are not suitable for citation in an encyclopaedia (wrt to medical facts). This page is not read by Wikipedia's readers - it isn't for them. It is for editors. Are you trying to help improve the quality of our sources, or defending popular journalism? Colin°Talk 17:11, 16 April 2007 (UTC)

Colin, I'll restate that. If the reviewers of the major journals, who are academic experts in the field, recommend a popular book as a reliable source, I would accept that book as a reliable source for Wikipedia. Why not? Nbauman 18:38, 17 April 2007 (UTC)
They aren't "recommend[ing] a popular book as a reliable source" for medical facts in an encyclopaedia. They are saying that a book is to be recommended for reading. Such books might be suitable sources for Wikipedia (particularly those that have footnotes, citations and a scholarly comprehensive style). There are usually sources that would be a better choice. However, I return to newspapers, which is our source of dispute. Newspapers are good for news; old newspapers are good for old news. If my doctor offered advice based on what he read in the Daily Mail that day, I'd report him to the authorities. Wikipedia should be no different. Colin°Talk 22:01, 17 April 2007 (UTC)
OK, here's a newspaper article that got the clinical facts more accurately than the peer-reviewed medical journals did. If your doctor had depended on the Wall Street Journal rather than the New England Journal of Medicine in considering whether to prescribe Ketek, he might have saved your life. Saving patients' lives is after all one of the purposes of medicine. You might rather die than have your doctor use information from newspapers rather than peer-reviewed journals, and I admire you for sticking to principles, but it seems a bit pedantic.
[4]Infected Data: Fraud, Errors Taint Key Study Of Widely Used Sanofi Drug; Despite Some Faked Results, FDA Approves Antibiotic; One Doctor's Cocaine Use; Company Defends Safety, By ANNA WILDE MATHEWS, Wall Street Journal, May 1, 2006.
... Now documents including internal Aventis emails reviewed by The Wall Street Journal are raising questions about a key clinical trial -- called study 3014 -- of more than 24,000 people that the company submitted to the FDA seeking approval for the drug.
The doctor who treated the most patients in the study, Maria "Anne" Kirkman Campbell, is in federal prison after pleading guilty to defrauding Aventis and others. An indictment says Dr. Campbell fabricated data she sent to the company. The documents show that Aventis was worried about Dr. Campbell early in study 3014 but didn't tell the FDA until the agency's own inspectors discovered the problem independently....
The full extent of the study's problems has never been made public. Its results were cited last month in an article in the New England Journal of Medicine that suggested Ketek is as safe as other antibiotics. Five of the six authors of that article disclosed that they received consulting fees from Sanofi-Aventis, and the sixth was an Aventis employee at the time of the study. Nbauman 02:15, 20 April 2007 (UTC)

[edit] Avoid citing the popular press

The section, In science, avoid citing the popular press is complete personal opinion, completely unsourced, completely overgeneralized, and completely wrong. For example:

[5]Annals of Medicine: The Bell Curve; What happens when patients find out how good their doctors really are? by Atul Gawande, The New Yorker, Dec. 6, 2004

Gawande is an MD, and in addition to the New Yorker he writes for the New England Journal of Medicine. Does the author of this section believe that Gawande does not cover science well when he writes for the New Yorker, but does cover science well when he writes for the NEJM? Similarly, Gina Kolata, a PhD, used to write for Science magazine before she moved to the New York Times. I could give many similar examples.

Many articles in the popular press have identified problems in medicine that have been ignored by the peer-reviewed literature, for example defective heart defibrillators, or financial conflicts of interest in the committees that set guidelines and recommend drugs. Peer-reviewed journals often cite newspapers in the footnotes.

The writer of this section does not seem to have consisdered that scientific results are normally released first as presentations at scientific meetings before they are peer-reviewed, and that is where the newspapers and magazines find out about them.

I believe that every article in the popular press should be evaluated on its own merits. You can't replace critical evaluation with a rule of thumb like 'The popular press generally does not cover science well."

This section should be completely rewritten. Nbauman 17:24, 14 April 2007 (UTC)

OK, as an example, if that author also writes for peer-reviewed journals, we should 1) be able to quote him from peer-reviewed work, and 2) be able to justify that the particular author is a reliable source, even if the popular press normally isn't. Remember, reliability of sources, like everything else on Wiki, is subject to consensus. Again, I think we can find compromise wording to the effect that there may be times when a reliable source is quoted in the popular press, but we should consult medical consensus and make sure they got it right. SandyGeorgia (Talk) 18:02, 14 April 2007 (UTC)
I think the consensus statement should start out by saying that the accuracy and reliability of the popular press varies greatly, and the accuracy and reliability of articles may vary within one publication. The popular press can be helpful in finding and organizing information, but everything in the popular press is subject to verification from the peer-reviewed literature. Still, the popular press sometimes reports important information that the peer-reviewed literature does not.
There are library science publications, such as Magazines for Libraries, that evaluate the popular press.
The popular press exposed the dangers of Vioxx before the NEJM did. The Wall Street Journal recently exposed a conflict of interest at the NEJM in which the editor who wrote an editorial supporting a policy of the National Kidney Foundation was on the board of the Boston chapter of the NKF, as I recall.
You just have to examine the facts on an case-by-case basis, and sometimes all you can do is give the opposing viewpoints. Nbauman 18:39, 14 April 2007 (UTC)
Well, you got me on the Vioxx. hits much too close to home :'-( Colin is a better writer than I am, so I'll stay out of tweaking the wording, and leave that to the two of you, but I agree with some of your points now. We just need to word it in a way that doesn't open the door too wide. SandyGeorgia (Talk) 18:56, 14 April 2007 (UTC)
I'm going to think a bit more about this one before changing any text. I'll read the Bell Curve thing later too. Hope someone else joins the debate. The current Newspaper's section reads more like a polemic than a guideline. Remember the context: medical facts. This isn't a guideline for current-affairs, history, biography, sports, etc. I'd love to see an example where citing a newspaper on a medical issue was better than citing an alternative source.
Your arguments that some articles in newspapers are OK can be extended to blogs, personal web sites, drug-company adverts, press releases, vanity press and all the other things that we guide against. The "is an MD" attribute doesn't make someone the holder of the truth. There are plenty nutty MDs out there.
You are in the privileged position of being able to judge whether an individual source (on certain topics) is good or not. Many of our editors, and almost all of our readers are not. This is why we have WP:V and WP:RS. The "truth" is secondary on WP.
WP should never use investigative reporting as a source. Newspapers get it wrong at least as often as right. They like to boast when right. Such reporting has only two editorial controls: "are our lawyers happy" and "will it sell papers". We all know of medical campaigns by newspapers that make the professionals despair. Wikipedia is in no rush. Better to report well established facts than up-to-date nonsense.
Many editors come to WP in order to evangelise the world about their treatment or their conspiracy theories as to the cause of illness. You only have to look at the more controversial sections of the autism topics on Wikipedia to find that such editors love quoting the tabloids.
You mention (in the guideline and above) that peer-reviewed journals "often" cite the popular press. Really? What percentage makes "often"? Are we talking about medical journals?
The following gets me worried:
"The writer of this section does not seem to have consisdered that scientific results are normally released first as presentations at scientific meetings before they are peer-reviewed, and that is where the newspapers and magazines find out about them."
This has been considered. The section on secondary sources says "Journalists writing in the popular press, and marketing departments who issue press releases tend to write poor secondary source material."
The sort of press-releases that hospitals/labs do, which are written to stimulate funding and press interest rather than advance medical science, are the last thing WP should be using as a source. And when newspapers get hold of these, they either regurgitate verbatim and uncritically, or else they mess with it. I'll try to find an example if you like. The latest cutting edge unproven research is really not what an "encyclopaedia" is about IMO.
Sorry about the rant. Your userpage says your are a professional writer. Then I'd love to know your opinion and get your support for WP:MEDMOS, which is being discussed nearby! Colin°Talk 20:06, 14 April 2007 (UTC)


Librarians have lists of magazines that they have determined to be reliable, with the caveat that the ultimate responsibility for making a decision rests with the reader. One of the standard reference works, for example, is Magazines for Libraries. Some magazines are more reliable than others. The New Yorker is consistently reliable. So is the news section of the Wall Street Journal, though other sections, like the editorial and personal health columns, are not. There's no simple guideline that anyone could apply to determine whether a source is reliable. The only answer is that popular magazines are diverse, their quality varies, and they are subject to verification like anything else.
As for opening the door to blogs -- that's a problem, but some blogs are reliable sources. There should be a burden of proof that they're reliable, but if they meet that burden, OK. Best example I can think of is CancerGuide: Steve Dunn's Cancer Information Page (which BTW has a good discussion of how to evaluate reliable sources of information, from bench to clinical trial).
You wanted an example of a newspaper that cited the popular press. Here's one -- footnote 2. The academic publications didn't even cover this until the New York Times broke the story. The issue here was that Guidant knew their ICDs were short-circuiting, but kept selling the existing stock even though they had solved the problem with newer models. (I still don't understand why.) The Guidant entry in Wikipedia doesn't seem to have cited them. I'm excerpting these articles since you may not have subcriptions.


[6]The Controversy over Guidant's Implantable Defibrillators,
Steinbrook R,
N Engl J Med 2005; 353:221-224, Jul 21, 2005. Perspective
On October 4, 2001, Joshua Oukrop, a Minnesota teenager with hypertrophic cardiomyopathy and a high risk of sudden death from ventricular fibrillation, received an implantable cardioverter–defibrillator (ICD). The device was a Ventak Prizm 2 DR Model 1861 manufactured by Guidant (Indianapolis). After it was implanted, Oukrop's physicians at the Minneapolis Heart Institute Foundation checked it every three months (most recently on January 31, 2005) and found no problems.
On March 14, 2005, Oukrop, then a 21-year-old college student, collapsed and died in a remote area of southeastern Utah during a spring-break bicycling trip with his girlfriend.1,2 An autopsy revealed no clinically significant pathology beyond his massive left ventricular hypertrophy. His physicians were stunned by his death. ICDs have been shown to be almost invariably successful in preventing sudden death in young patients with hypertrophic cardiomyopathy, as long as they do not have end-stage heart failure — which Oukrop did not.3 When the manufacturer analyzed his ICD, it determined that the device had short-circuited internally while trying to deliver high-voltage therapy and had been permanently disabled (see diagram). Moreover, its memory had been destroyed, making the time of failure impossible to pinpoint....
2. Meier B. Maker of heart device kept flaw from doctors. New York Times. May 24,2005:A1.


[7]Maker of Heart Device Kept Flaw From Doctors
By BARRY MEIER, New York Times May 24, 2005
A medical device maker, the Guidant Corporation, did not tell doctors or patients for three years that a unit implanted in an estimated 24,000 people that is designed to shock a faltering heart contains a flaw that has caused a small number of those units to short-circuit and malfunction.
The Potential for Defibrillator Failure
The matter has come to light after the death of a 21-year-old college student from Minnesota, Joshua Oukrop, with a genetic heart disease. Guidant acknowledges that his device, known as a defibrillator, short-circuited. The young man was in Moab, Utah, on a spring break bicycling trip in March with his girlfriend when he complained of fatigue. He then fell to the ground and died of cardiac arrest.
Guidant subsequently told his doctors that it was aware of 25 other cases in which the defibrillator, a Ventak Prizm 2 Model 1861, had been affected by the same flaw. Guidant said it had changed its manufacturing processes three years ago to fix the problem. The physicians say that had they known earlier, they would have replaced the unit in their patient because he was at high risk of sudden death. His death is the only one known....
Nbauman 21:16, 14 April 2007 (UTC)
But there's a theme to your examples, Nbauman, which may help us solve this dilemma. They are tied to current events. The news media stories came about because of current events, as opposed to being reports per se about the conditions or studies about the conditions. I may not trust most newspapers to accurately tell me about the most recent studies on hypertrophic cardiomyopathy, but maybe I can trust some newspapers to report a story about a current event involving that condition correctly. SandyGeorgia (Talk) 21:30, 14 April 2007 (UTC)

[edit] The Bell Curve

I've not finished reading this but already spotted something that highlights why newspapers make poor secondary sources. Page two discusses various operations and the variable success rate amongst surgeons. One example given:

A Scottish study of patients with treatable colon cancer found that the ten-year survival rate ranged from a high of sixty-three per cent to a low of twenty per cent, depending on the surgeon.

The article doesn't fully cite the study, which is typical and understandable (though perhaps not for the online edition, which has no space concerns). So we don't know:

  1. Why, on an article in an American magazine, with mostly American examples and statistics, does the author pick a Scottish study? Are they desperate to find an example to back their case? Or perhaps this study is internationally famous?
  2. How big was the study? How many patients? How many hospitals? Did it just cover NHS hospitals or private ones too?
  3. How was the study conducted?
  4. What threshold did they use for "treatable"? Did all the hospitals use the same threshold when choosing patients?
  5. What other factors influenced patient outcome? Perhaps patients living in poor areas did worse than richer ones and this affected the local hospitals they went to?
  6. Perhaps there was a hospital variation, not just the surgeon. A patient's recovery depends on many professionals' input, not just the surgeon.
  7. Did they have other non-surgical therapies too?
  8. The article makes a case that the range of success follows a bell curve. But this example only quotes two extremes. We have no idea what shape graph this study produced.

And so on. Most "for-professionals" articles would give more information than this, plus a citation. Without this traceability from Wikipedia to secondary source to primary source, our readers are limited in how much they can learn should they ask questions concerning the reliability of the data and how the author chooses to use it to make their case. The best popular science/medical books provide citations. So should online newspapers. Colin°Talk 23:09, 14 April 2007 (UTC)

Gawande was citing a classic, frequently-cited study that every oncologist would know. [8]McArdle et al, BMJ 1991;302:1501-5, Impact of variability among surgeons on postoperative morbidity and mortality and ultimate survival. University Department of Surgery, Royal Infirmary, Glasgow.Nbauman 06:03, 15 April 2007 (UTC)

Every oncologist might think "Ah, she's citing McArdle and Hole 1991" but almost none of our readers would. I was going to try to look up the paper myself but you've saved me the trouble. Unfortunately, it only goes to strengthen the argument that the primary source should be cited by WP rather than the newspaper (which may be less than helpful in finding it). For example, I can now cite:

which fortunately has the full text available free online. Even from the abstract, our reader can tell more than the newspaper provided. We can also see there were two follow-up letters, one critical and one supportive. PubMed also tells me that McArdle and Hole continued their research. Which is just as well: these were "patients with colorectal cancer presenting over the six years from 1974 to 1979". The quality of an operation performed 30 years ago is of diminishing interest to those going under (or holding) the knife today. They published a follow-up paper in 2002 (with a much larger number of patients, operated over the years 1991 and 1994) that confirmed that variability amongst surgeons was still a problem:

  • McArdle C, Hole D (2002). "Outcome following surgery for colorectal cancer: analysis by hospital after adjustment for case-mix and deprivation". Br J Cancer 86 (3): 331-5. PMID 11875693. 

So it could be argued that the first might be a historical classic, but the second is of more relevance today. They have also answered some of my questions about poor patients doing less well:

  • Hole D, McArdle C (2002). "Impact of socioeconomic deprivation on outcome after surgery for colorectal cancer". Br J Surg 89 (5): 586-90. PMID 11972548. 

I see they have refined their conclusions about surgeon variability, by showing that surgeon speciality is a better guide than just volume of work:

  • McArdle C, Hole D (2004). "Influence of volume and specialization on survival following surgery for colorectal cancer". Br J Surg 91 (5): 610-7. PMID 15122614. 

I could go on (for example, to check that those who cite this paper do so favourably) but in just a couple of minutes, I've found so much more high quality source material to help improve a WP article on either colorectal cancer or issues of surgeon competence. Colin°Talk 07:00, 15 April 2007 (UTC)

Let's return to the original question. Is the New Yorker a reliable source of medical information or not? Nbauman 13:47, 15 April 2007 (UTC)
Back to what I said above; there are some things that can possibly be cited to some of the news media (I've done it in Sociological and cultural aspects of Tourette syndrome), but a medical, peer-reviewed source is always preferred, as demonstrated in Colin's example. The news media rarely gives us the full story. Let's make sure our wording gives a strong preference to peer-reviewed sources. I believe (haven't checked) we don't say never cite the popular press; we urge caution. Perhaps we can expand/contract that to say they shouldn't be preferred over the highest-quality, peer-reviewed medical sources, but do have some usefulness. I think (?) your concern is that we may be "bashing" the popular press; perhaps we can better flesh that out. SandyGeorgia (Talk)
Right, I don't want to bash the popular press. Most users of Wikipedia use the popular press (many academic doctors read the popular press, as demonstrated from their footnotes), so we should help them place the popular press in its proper context. I think the context is that their accuracy and reiability varies greatly. Some magazines are accurate enough that we can presume them to be accurate, but they're always open to challenge, and peer reviewed literature usually (but not always) trumps the popular press. There are evaluations of the popular press, in the links I added under Newspapers. Nbauman 18:21, 15 April 2007 (UTC)

Gawande's work for NEJM is peer-reviewed. His journalism is not. In general, peer-reviewed science is the preferred source for any medical content, with all other sources inferior to it. Within peer-reviewed science, I think we should adopt EBM grading. A meta-analysis or systematic review is much more powerful as a source than individual trials, case-control studies, case series or case reports. JFW | T@lk 20:14, 15 April 2007 (UTC)

JFW, can you help work on the wording with Colin here? SandyGeorgia (Talk) 20:24, 15 April 2007 (UTC)
Some of those ideas are already very briefly mentioned but not in such an explicit order. JFW is much, much more knowledgeable on this front than me so I'd rather defer to him. I can tidy the English if required :-). I'd really like JFW to contribute to this, since he's often the one defending these things on the project or article talk pages. Colin°Talk 20:47, 15 Apri
I don't know. Is the Perspectives section of the NEJM peer-reviewed? Nbauman 00:05, 16 April 2007 (UTC)
That's a good point. Not all forms of writing in these journals get the same treatment. For example, an obituary may be no better checked than one in a newspaper. There are several aspects to judging the quality of a source, not just the journal title or type. Colin°Talk 07:38, 16 April 2007 (UTC)

[edit] TV, radio

Re popular press, now that TV and radio news programs have websites, I see a need for some guidance re citing them. See for example fetus in fetu, where one editor cites ABC and MSNBC news as sources of medical data (incidence, treatment). --Una Smith 15:20, 9 July 2007 (UTC)

What wording do you propose (can we make it more general, and not focus on any specific aspect of the popular press)? SandyGeorgia (Talk) 15:26, 9 July 2007 (UTC)

[edit] Core journals

This section needs a paragraph explaining that niche journals have low impact factor regardless of quality, because their readership is small; niche journals can be evaluated (and to some extent compared to "core" journals) based on their average article halflife, meaning the number of years over which an article is cited. --Una Smith 15:28, 9 July 2007 (UTC)

If at all possible, avoid promoting the idea of rating an article by the author's authority: where they work, their title, their rank. Judge the person by the quality of the work, not vice versa. --Una Smith 15:28, 9 July 2007 (UTC)

[edit] Proposed Guideline?

This page was set up as a Proposed Guideline. It would appear that WP only accepts such a proposal for a finite time before marking it Historical or Rejected. Can we decide what we are going to do with it? My gut feeling is that there currently isn't enough traffic or discussion on this for it to move forward quickly enough to become a formal guideline before someone retires it again. If someone else wants to beat a drum to round up some contributors, then great. I was hoping that we might get contributions from editors with experience writing or training reading medical articles. I do believe the project needs these guidelines

  1. To have somewhere to point at when the issue arises.
  2. To have a central place to discuss what kind of sources are good for medical info and how they should properly be used.

But it may be that those needs can be met by taking the banner off and leaving it as an informal guideline in Project space. Thoughts? Colin°Talk 11:35, 15 May 2007 (UTC)

The editor who tagged it rejected tagged dozens at the same time; I doubt he read the ongoing discussion at WP:MEDMOS. I think we can push for guideline status, with just a bit more review. Since we were busy on MEDMOS, we haven't really pushed. SandyGeorgia (Talk) 11:45, 15 May 2007 (UTC)
If I may chime in... proposals tend to be marked "historical" if there's no active debate going on. This is a matter of convenience, not a fixed difference. In particular, if active debate starts once more (e.g. through advertising the page) then it's active and no longer historical. I see no substantial dissent on this talk page, although I lack the expertise in medicine to vouch for the quality of the page. If a few medically-inclined people can confirm that it's (1) accurate and (2) useful, then it would make a worthwhile guideline. Guidelines are not a big deal and are certainly allowed to have exceptions. >Radiant< 11:55, 15 May 2007 (UTC)
(A) If there is discussion regarding development of this guideline it should be reflected here
(B) There is insufficient breadth of contribution here to demonstrate a consensus for acceptance
(C) The onus to demonstrate consensus is on the proponents and in default a proposal is rejected without demonstrable consensus
(D) Yes, this was included in a cleanout of dead proposals, and for now I remove my advocacy of rejection. --Kevin Murray 12:07, 15 May 2007 (UTC)
Our efforts (and discussion) had been focused first at WP:MEDMOS; we'll get this moving again. SandyGeorgia (Talk) 12:10, 15 May 2007 (UTC)

[edit] Not ready for guidelines

I don't think this guideline is ready to be adopted. Much of it is unsourced, unsupported personal opinion. For example, "The popular press generally does not cover science well." Who says so? What is their evidence? Isn't this an overgeneralization? In some cases, the Wall Street Journal has turned out to be more reliable than the New England Journal of Medicine.

Why doesn't the entry cite the extensive literature on how the popular press covers science (which finds that the quality of coverage varies from very good to very bad)? What about the library literature, such as Magazines for Libraries?

A fundamental problem is that this is just a list of sources and judgments about them. It wouldn't be much help in resolving real disputes that go on in real articles. Define the problem. I would suggest that you look at some actual controversial medical entries, and examine the disputes that come up over reliable sources, etc. Look at Dichloroacetic acid, or Diabetes.

People do cite the popular press all the time. What should we do about it? Most of the peer-reviewed literature isn't available free on the Internet, so it isn't verifiable to someone who doesn't have a subscription. What do we do about that?

What are the disputes and how should they be resolved, in terms of reliable sources? Nbauman 14:26, 15 May 2007 (UTC)


Maybe the proponents could provide some examples of articles which would be removed if this guideline was adopted and why they are inappropriate for inclusion at WP. --Kevin Murray 14:32, 15 May 2007 (UTC)

Kevin, not sure what you mean by articles which would be removed; we're talking about sources here ? SandyGeorgia (Talk) 16:01, 15 May 2007 (UTC)

I agree 100% that this is not ready to become a guideline right now. That's why I now think our use of "Proposed" may be a little premature. I had thought the label was OK for saying "I propose we have some guidelines on this. Here's a start I've made...". But it is being interpreted as "Here's a set of guidelines I propose. Discuss...". I don't really mind what the banner says (but not "rejected", please) or if we don't have one. It would be nice to have some kind header/intro that said:

Wikiproject Medicine would like to propose some guidelines on choosing and using sources for medical articles. These guidelines are at an early stage so all contributions are welcome. Please be bold and be willing to discuss.

I think Nbauman makes some good points, but WP guidelines do not need to be sourced. Opinion is fine if there is consensus.

I'm finding some of the newspaper arguments are starting to repeat so would welcome input from others. Can we please stay focused that these are guidelines for medical facts. When has The Wall Street Journal ever been "more reliable" for medical facts than the NEJM? I'm not talking about breaking some medical scandal a few weeks early. I've just tried searching their online site for medical info and have been unable to find anything other than articles about how some drug approval or loss of patent is affecting some company's share price, or a paragraph on new research opportunities (that affect a company's share price). Colin°Talk 15:21, 15 May 2007 (UTC)

The statement "In some cases, ... has turned out to be more reliable than" just doesn't work. I'm sure you can find cases when it was "more accurate than" for a given topic and moment in time. But "reliable" implies one can regularly, not just occasionally, depend on it. If this was an essay on "Accurate sources" then a survey of the "extensive literature" on different sources and their quality might make an interesting read.

The other aspect we need to consider is "useful". I think the example on the Bell Curve above showed that newspapers aren't as useful a source as a journal, even if the information is technically accurate.

I'll have a look at your other points later to see what I can find. Colin°Talk 15:58, 15 May 2007 (UTC)

[edit] Popular press example

Here's an example of using the popular press, which I cleaned up using peer-reviewed or medical sources just last week — from an area I'm familiar with.

Note the headline — gene found !! This is what the popular press does.

When in fact ... The peer reviewed sources are available, and the finding is reported in a way that is more "scientifically" and "medically" correct. The BBC merely parroted some portions of the Duke Medical News, while adding nothing clarifying or illuminating.

Note the more cautious and accurate headline in the Duke Medical press release, and The researchers estimate that the SLITRK1 mutations account for 5 percent of trichotillomania cases. This gene is not significant in and of itself (it is not *THE* gene that has been discovered as some earth-shattering event), as much as it provides a vehicle for future research directions. There is no need to cite the "hyped" BBC version, when the peer-reviewed article can be found in a medical library (and we don't choose our sources based on whether they are easily available online, free or not - we choose the best and highest-quality sources period, even if they're not available online, which should be the peer-reviewed medical literature).

Further, if the issue of citing the popular press is the only problem with these guidelines, we can work on the wording. But, whenever peer-reviewed sources are available, they could at least be preferred over the popular press, which tends to hype results and take them out of context or proportion. SandyGeorgia (Talk) 16:12, 15 May 2007 (UTC)

Ah Sandy, I'm very disappointed in you ;-) You didn't spot the spelling mistake in the press release that has interesting consequences. The first time the press release mentioned the gene, they spell it SLITKR1, not SLITRK1. Now Google for [Trichotillomania SLITKR1]. You will find all the news sources that uncritically used the press release (267 hits). If you Google for [Trichotillomania SLITRK1 -SLITKR1], you wont find any popular news articles (except a couple of very short ones) but a fair amount of research.
The press release is also somewhat flawed, though not as much as the popular press version. For example, at the start they say scientists "have identified gene mutations that cause trichotillomania" but much later weaken this to "The SLITRK1 gene could be among many other genes that are likely to interact with each other and environmental factors to trigger trichotillomania". They don't say how many people have SLITRK1 mutations without trichotillomania, for example. Is this a "cause" or just a "factor"? They also estimate it accounts for 5% of cases but the press release doesn't explain what gives them that idea. I thought the BBC were dumbing down with talk of "faulty wiring" but that came from the press release! Lots of handwaving going on here :-) Colin°Talk 17:18, 15 May 2007 (UTC)
Right. The press did the same thing when this gene was implicated in Tourette's, and you had to read the actual journal report to sort it all out. The popular press cannot be counted on to report this sort of thing correctly. I haven't read the full journal report in this case, because I know it's the same kind of deal as in the TS connection. This gene may be a factor for a small subset of patients, which makes it interesting for research, rather than some kind of breakthrough. SandyGeorgia (Talk) 17:26, 15 May 2007 (UTC)
Re the WSJ sometimes being more accurate than the NEJM, I read both regularly and I could find many examples. I already mentioned COX-2 inhibitors. A recent one is Ketek -- the NEJM published an article citing an industry-sponsored study, but the WSJ pointed out that the doctor who recruited the largest number of patients in that study was indicted after an FDA investigation, pleaded guilty to fraud for fabricating patients and falsifying patient records in the study, and is now in jail. If you read the Perspectives section in the NEJM, you'll see news-style articles that report on these problems, though the NEJM is not always forthright on their own role. Nbauman 18:15, 15 May 2007 (UTC)
But, again, this was already covered in our discussion above. This isn't reporting about the "medicine"; it's reporting about a current even surrounding the medical event, in which case citing the popular press makes sense. Maybe we need to reword to clarify the distinction? When we went over this (above), it seemed we agreed. SandyGeorgia (Talk) 19:33, 15 May 2007 (UTC)
The question in that case was whether there were scientifically valid studies to support the safety and efficacy of Ketek. The NEJM claimed, without qualification, that there were. The WSJ reported that the studies did not follow scientific protocol and were not valid. The WSJ reported unpublished FDA reports, which they got through FOI requests and leaks, about liver failure from Ketek. That's medicine.
Michael Fiore was on the Public Health Service panel whose guidelines said that smokers couldn't quit without nicotine patches, gum and other substitutes, and published articles saying that in peer reviewed journals. The WSJ reported that Fiore got $1 million from GlaxoSmithKline which makes Nicorette gum, and manufacturers of other substitutes -- without disclosing his financial ties. The WSJ quoted other experts and cited studies which found that patches, gum, etc. offered no benefit. That's medicine.
Here you have peer-reviewed journals publishing articles by a paid consultant to drug companies, who recommends his company's drug without disclosing his financial interest. A newspaper discloses those interests, reports other studies that say the drug doesn't do any good, and interviews medical experts who confirm that. The question of whether nicotine patches help people stop smoking is a question of medicine. Which source is more reliable?
You cannot make an unqualified statement that newspapers or popular magazines are not as reliable as peer-reviewed journals. Nbauman 20:21, 15 May 2007 (UTC)
Scandal, scandal, scandal. That's all you, and the newspapers, seem to be bothered about. Unless there's a "story" to be told, the press just aren't interested. What about all the drugs that work and were well researched? What about the stories in the UK press about the latest cancer "treatments" being denied to patients on the NHS that when you investigate them, are perhaps for the very good reason that there is no evidence that they work, or work any better than existing, cheaper treatments. The press get these stories wrong at least as often as right. You still haven't proven they are "reliable" at all. Only that, occasionally, they get something right and perhaps prove others wrong. When you say that the "WSJ quoted other experts and cited studies which found that patches, gum, etc. offered no benefit" it would be doing so selectively in order to make its case and emphasise the scandal. The journalist would certainly have ignored any studies that disagreed with their case. I accept and understand that. It is not medicine, it is journalism. It has a useful place in our world, but its place in an encyclopaedia is (as Sandy says) when documenting current/historical events.
Only a very tiny percentage of drugs and medicine interest the press. If it is good news, it is over-sold. If it is bad news it is the worst scandal ever. An encyclopaedia does not need to be at the cutting edge. We can wait for things to be proven, and if some of that "proof" was wrong or fraudulent, we can wait for judges and experts to agree it was wrong. Colin°Talk 22:12, 15 May 2007 (UTC)
You're simply asserting your own personal opinion, without providing any evidence. I'm giving multiple specific examples, and also citing academic research, such as Health News Review, where journalists and doctors review and evaluate the news coverage -- and come to different conclusions than you do. Why don't you consider the possibility that many doctors who specialize in patient education, and disagree with you, may be right?
You state that "The journalist would certainly have ignored any studies that disagreed with their case" -- but this is not true, and you would have seen that if you had read the article itself. The WSJ always gets both sides of the story, as it did in this case. You're the one who is ignoring facts that disagree with your case. Nbauman 12:41, 16 May 2007 (UTC)
We've already been over examples from all sides, and this is going in circles. You've given examples of current events that some newspapers got right. So that we can stop going in circles, can you propose some wording that 1) gives preference to peer-reviewed sources over the popular press when they are available (see the example I gave), and still allows for popular press reporting of current events, while avoiding "recentism"? We need proposed wording that recognizes the value of peer-review, and the mistakes that can be made in the lay press. SandyGeorgia (Talk) 15:04, 16 May 2007 (UTC)

Yes, that's what I'm trying to do. I only object to the oversimplified, unsupported statement that (1) If it appears in the popular press it's not reliable and (2) if it appears in the peer-reviewed literature it is reliable.

My position is that the popular press varies in reliability. I gave you links to a web site run by doctors and journalists, in which doctors review and evaluate the reliability of articles in the popular press. I also cited reference books that librarians use, such as Magazines for Libraries, that evaluate the reliability of popular magazines. Some popular publications and newspapers are more reliable than others.

Peer-reviewed journals are usually more reliable, but not all of them. I liked to a publication, the Brandon-Hill list, that lists the most reliable peer-reviewed medical publications. Some peer-reviewed publications are financed by drug companies, or medical device companies, and publish "peer-reviewed" articles that support the use of their products. So some peer reviwed journals are more reliable than others.

I would suggest that the guidelines include language like that above.

In general, good peer-reviewed publications are more reliable than the newspapers or popular press. But there are lots of exceptions. Even the peer-reviewed journals, like The Lancet, will publish articles that they know are wrong, because they want to get the argument out for debate, as they did with that article on rats who ate genetically modified potatoes. The BMJ (I think) published an article on mercury preservatives in vaccines which they and every legitimate doctor have repudiated, but people keep quoting it. Nbauman 21:26, 16 May 2007 (UTC)

[edit] Classification and grading of clinical evidence

Hello. Are the AHRQ guidelines on grading evidence too technical for inclusion here? I'd really like to see a summary of the grading as outlined in evidence-based medicine included here - I think it's important, no? Just a table like this:

  • Ia: evidence from meta-analysis of randomised controlled trials (RCTs).
  • Ib: evidence from at least one RCT.
  • IIa: evidence from at least one well-designed controlled study without randomisation.
  • IIb: evidence from at least one other type of well-designed quasi-experimental study.
  • III: evidence from well-designed non-experimental descriptive studies, such as comparative studies, correlation studies, and case control studies.
  • IV: evidence from expert committee reports or opinions and/or clinical experience of respected authorities.
Grade Evidence Description
A Ia, Ib Requires at least one RCT as part of the body of literature of overall good quality and consistency addressing the specific recommendation
B IIa, IIb, III Requires availability of well-conducted clinical studies but no RCTs on the topic of recommendation
C IV Requires evidence from expert committee reports or opinions and/or clinical experience of respected authorities. Indicates absence of directly applicable studies of good quality.

Would do the trick. Thoughts? Nmg20 17:37, 15 May 2007 (UTC)

I think that would be a good and even necessary addition. You can't write about medicine if you don't understand this. Nbauman 18:17, 15 May 2007 (UTC)

Please add to the page, we can tidy/refine later. There is more than one way of grading sources. The current headings (Periodicals, Books, Online) could perhaps be demoted to 2nd level under a new "Media types" heading (or similar wording). Then we could have another top level heading for "Research types", for example.

We should link to Trish Greenhalgh's "How to Read a Paper" Series. If people read that, they'd have a good idea of different study types. Colin°Talk 22:20, 15 May 2007 (UTC)

I've added something along these lines, although I haven't added the link to the "How to Read a Paper" series, which should certainly go in. I wasn't feeling very inspired when I made the changes, though, and it certainly needs proofing by a couple of other editors if anyone has time... Nmg20 01:59, 18 May 2007 (UTC)

[edit] Question about conference proceedings

I find that it is often easier to access reprints of conference proceedings online than complete journal articles. I'm not sure if that is only true for myself as a veterinarian or if it also applies to M.D.s. Where do these proceedings fall in the matter of reliability, in this project's opinon? Note that these are major conferences, with well-respected lecturers, and I'm only referring to reviews of topics, not new research. Thanks. --Joelmills 03:12, 25 June 2007 (UTC)

Still easier: don't bother citing anything. (That was a joke.) I cite such sources only as a last recourse, and then only if I know the article actually was presented at the conference. Sometimes the proceedings are published before the conference, and then at the conference the article is retracted! This is more common when the proceeding volume consists of abstracts or short articles that are little more than abstracts. Some proceedings volumes are peer reviewed and/or of the highest quality, but they are in the minority, and you really have to know the specific research community to know which proceedings volumes are top-notch and which ones are not. It isn't enough to go by series title, because this can change from year to year, depending on who is the editor. --Una Smith 15:40, 9 July 2007 (UTC)

[edit] Proposed changes

In the spirit of being bold, I've made a few changes to the proposal. Here is a summary diff. Feel free to revert them if they seem redundant or inappropriate. In general, I'd like to be a little more explicit on the fact that primary sources (journal articles reporting original findings) are a welcome and even necessary part of medical articles, but that the interpretation of such research must hew carefully to that provided in reliable secondary sources (reviews/textbooks). I've seen significant issues with editors citing a number of basic-science journal articles and then leaping to a totally off-the-wall conclusion, which is then defended as "cited content".

Another issue is articles on supposed medical conditions which have never been reported or recognized by any medical authority (see mucoid plaque). I would favor including something in the guideline along the lines of, "If a purported medical condition, test, or treatment has been described and evaluated by the medical community, then it should be easy to cite reliable sources on the subject. In the absence of such sources, topics should not be presented as if they are accepted by the medical community." But perhaps this is overstepping the bounds of this proposed guideline. MastCell Talk 17:04, 25 June 2007 (UTC)

A related example is pyroluria. SandyGeorgia (Talk) 17:07, 25 June 2007 (UTC)
Yes, good example. In both cases I think it's important to place the burden of proof where it belongs (i.e. show us that this is a real, accepted medical condition) rather than being in the position of trying to prove a negative. Whether this guideline is the place to do so, I don't know. MastCell Talk 17:10, 25 June 2007 (UTC)
I very much agree with a statement like that being included into the proposal. Also, there are ideas of diseases/conditions made as a statement of fact, even though the person espousing such ideas freely admit that they go against the medical community. In other words, a person states something like, "the medical community states this, but that is wrong or baseless because of this fact." That does not help in making articles reliable. Perhaps the sentence could read something like, ..."In the absence of such sources, topics should not be presented as accepted by the medical community or as a statement of fact." Or some variant of that. Either way, no matter what "facts" someone presents, if it goes against the vast medical consensus, then those views should not be presented as "facts". Of course, truly contested material should be presented as such. Perhaps a sentence should be inserted saying something similar if one is not included already. - Dozenist talk 18:29, 25 June 2007 (UTC)

[edit] PubMed vs. secondary sources

Discussion moved from Wikipedia talk:WikiProject Clinical medicine:

WP:MEDRS seems to be at odds with WP:MEDMOS; MEDMOS encourages the use of PubMed references, MEDRS implicitly discourages them.

WP:MEDRS states:

In general, Wikipedia's medical articles should use published reliable secondary sources whenever possible. Reliable primary sources may be used only with great care, because it's easy to misuse them. For that reason, edits that rely on primary sources should only make descriptive claims that can be checked by anyone without specialist knowledge. Any interpretation of primary source material requires a secondary source.

In my opinion:

  • The above (in WP:MEDRS) should be further qualified. Primary sources, IMHO, are accessible to an interested layperson, with the vast amount of credible medical information (e.g. Merck Manual, eMedicine, Medlineplus.org, |Canadian Health Network) out there and the strong base of Wikipedia articles that cover topics in medicine and experimental physiology.
  • Primary sources should be the key references-- secondary sources should be considered supplemental. Primary sources should be explained -- like any good secondary source for the lay public.
  • Good secondary sources base their info from primary sources. I think Wikipedia has enough people with expertise to deliver nuanced interpretations of primary sources that can compete handily with respected secondary sources.
  • Use of secondary sources from PubMed (i.e. review articles) should be encouraged.

I look forward to the discussion. My thoughts on this arose from this discussion-- and are related to changes to the McClintock effect article. Nephron  T|C 06:12, 24 June 2007 (UTC)

I wonder whether that's an old policy which is now outdated as the 'pedia continues to grow in depth. I don't worry about it myself and often use primary sources as do many others. Have a look at alot of FA nominees.cheers, Casliber (talk · contribs) 06:17, 24 June 2007 (UTC)
I didn't think scientific journals were necessarily considered primary sources. Mostly, it means that care should be exerted when using recent, sweepingly new results, but I've never seen people objecting to theuse of scientific journals on the base of them being "primary sources." I guess this is because no article can exist that is not based on some amount of existing work. I certainly agree that review articles/material should be encouraged, if only because that material is generally more directly accesible, and can provide of otherwise difficult to locate further material. The couple review sources I used for Verbascum thapsus provided most of the important structure for the article. Circeus 07:18, 24 June 2007 (UTC)
  • It seems like a non-sensical sentiment to dismiss pubmed indexed articles as preferred sources. I agree, however, that people who are unfamiliar with some areas will use refs. out of context to prove this or that. The expertise of editors in some of these specialty areas helps to filter the wheat from chafe. Secondary sources like standard medical textbooks, positions of health ministries (eg. FDA, health Canada) etc. can be used to bring contextDroliver 15:37, 24 June 2007 (UTC)
The danger with favoring, or relying on, primary sources is that you're depending on the expertise and insight of the editor to use and interpret them appropriately. For instance, a primary source stating that a compund shrinks human tumors engrafted in NOD/SCID mice may be presented as "XXX is highly effective against many human cancers." (Yes, I have been scarred by the DCA thing). In cases where 1 study has showed an effect, but dozens of others have confirmed that the effect does not exist, selective citation is a danger. I don't think we should deprecate primary sources (since they're so vital to explaining any medical topic), but I do think we should insist on something along the lines of, "The use of primary sources (e.g. journal articles) is encouraged on medical topics, but interpretations of these sources should hew carefully to that presented by the authors or by reliable secondary sources such as review articles and medical textbooks." This might discourage the inevitable idiosyncratic usage of primary sources while still encouraging their general inclusion. Thoughts? MastCell Talk 15:54, 24 June 2007 (UTC)
"The us of some types (e.g. scientific journal articles) of primary is encouraged" without reference to scientific topics in particular (why not physics, maths or astronomy??) is a better wording to me. Why don't you bring your suggestion to the talk page? It should be noted that by Wikipedia's own definitions, Medical journals articles are neither "original research" ("unpublished facts, arguments, concepts, statements, or theories" or "any unpublished analysis or synthesis of published material that appears to advance a position"), nor are they "unreliable" (reliable sources: "credible published materials with a reliable publication process"), so for the most part, these concerns are unfounded. I don't think they fit Wikipedia's definition of Primary sources either, which refers mostly to historical documents. Examples cited are: "archeological artifacts; photographs; newspaper accounts which contain first-hand material, rather than analysis or commentary of other material; historical documents such as diaries, census results, video or transcripts of surveillance, public hearings, trials, or interviews; tabulated results of surveys or questionnaires; written or recorded notes of laboratory and field experiments or observations;". Furthermore, an example of a secondary source is "[a]n historian's interpretation of the decline of the Roman Empire, or analysis of the historical Jesus," which can certainly be a "primary source" journal articles. Further removed sources, because of their accessibility, continue to have advantages, but it is a Good Thing to be able to credit the originators of novel ideas. Circeus 17:53, 24 June 2007 (UTC)
  • I'm a bit puzzled by the idea that the comments in MEDRS "implicitly discourage" PubMed references since PubMed isn't mentioned at all. PubMed indexes medical articles, which may be primary or secondary sources. They may be research, letters, biographies, book reviews, case notes, etc, etc. MEDRS is based on RS (a while back) and I believe the views on Primary and Secondary sources apply to medicine just as much as history or science, for example. Perhaps there is a middle ground were we encourage editors to cite primary (often seminal) papers inline but back this up with review articles/textbooks listed as References where they have been used by the editor to confirm his interpretation (if any) of the primary material.
I feel we should also say something to discourage citing a primary source to confirm a fact that the editor only knows through reading a secondary source. The citation may include the primary source but should also say "as cited by ..." to give details of the actual source used by the editor.
If you read MEDRS, you should see it (attempts to) define what are primary/secondary sources wrt medicine. There most certainly are primary sources in medicine.
I'm pleased that MEDRS is starting to be discussed. It is not ready to become a guideline without much further work and discussion from project members. Should this discussion be moved to MEDRS's talk page? Colin°Talk 18:21, 24 June 2007 (UTC)
Good idea... I thought of that after I'd already commented here. I'll move over there. MastCell Talk 19:40, 24 June 2007 (UTC)

I have been looking at WP:MEDRS, and I would be relieved to have some guidelines such as those that are listed. This would become very relevant to topics in dentistry, such as "new-and-improved" products but especially on fluoride and amalgam. It seems to me that, by far, the most important item in MEDRS is that an article must "present the prevailing medical or scientific consensus." Anything else placed in an article should be labeled as a minority view or one that is not accepted by the established consensus. As long as this principle is followed, then I do not foresee major reliability nor original research problems arising. Secondary sources can be encouraged in the guideline to make certain that medical/scientific consensus is presented, but I think the most important point to emphasize is that (regardless of the source) the content presented in the article, whether held by consensus or a minority viewpoint, must be presented as such. Saying all this, I hope this proposal can eventually be elevated to a guideline with a little work. - Dozenist talk 14:26, 25 June 2007 (UTC)

When dealing with referencing medical articles, I personally rely on the following hierarchy:
  1. High-quality review in a high-impact factor core medical journal (e.g. Lancet, NEJM)
  2. Several primary studies corrobating each others' results, or a good meta-analysis
  3. Primary studies without evidence of a trend/phenomenon - in these cases good consensus on a talkpage is very useful
  4. Websites/popular press articles etc
Notability of primary/original research should really be agreed upon by consensus on the talkpage. JFW | T@lk 18:56, 25 June 2007 (UTC)
I think JFW's hierarchy is sound & reasonable. It's fairly easy to demonstrate a general consensus from reliable sources which should trump out of context single references on things. An important adjunct to this list would be referring to the standard current issue medical or surgical textbooks which tend to caputure a general overview at the time of publication. In most areas of medicine, there is little to cause radical reinvention of a field due to publication lag time.
Wikipedia, being what it is, has served as an attractive source for agenda-driven editors on topics like autism, breast implants, gulf-war syndrome, psychotherapy, homosexuality, and others. Dealing with non-mainstream POV proves hard with many such topicsDroliver 14:14, 1 July 2007 (UTC)


[edit] Misuse of primary sources

Have a look at this old version of Green tea. The FDA rejection of health benefits can be found here, which, although in the form of a letter, is the result of a serious review of the available evidence. Look how it is dismissed:

"Contradicting the FDA, A 2006 study ..."

I don't know much about green tea or those studies, but heavyweight studies such as the FDA one should not generally be placed lower in the importance-hierarchy than individual research papers. It is a common misconception on WP that primary is better, probably due to the word's other uses in the English language. MEDRS must not give this impression. Colin°Talk 18:17, 25 June 2007 (UTC)

I've edited the relevant section to try to emphasize this point, which I completely agree with. I would also plan to add something along the lines of, "Individual primary sources should not be cited or juxtaposed to "debunk" the conclusions of reliable secondary sources, unless the primary source itself makes such a claim. Controversies or areas of uncertainty in medicine should be illustrated with secondary sources describing the varying viewpoints." MastCell Talk 19:52, 25 June 2007 (UTC)

[edit] Textbooks vs. Monographs

To my mind, "undergraduate medical textbooks" are absolutely appropriate as sources for medicine-related articles, and are likely to be more appropriate for an online encyclopaedia than postgraduate ones. I've reverted this change in the main article for now, but reckon it merits a discussion on here... Nmg20 (talk) 16:01, 17 November 2007 (UTC)

My recent change of text (which you reverted) was actually a self-correction. I had mistakenly used the word "textbook" when expanding this page back in Nov 2006. I had not intended to mean undergraduate, college or school books, which is what a textbook is. Such books are usually extremely general in nature (e.g., a whole book on neurology), usually have few references themselves (which makes them a dead-end as far as research is concerned), contain irrelevant question/answer sections for student study, and are not always written by experts in the field (but rather, by experts in teaching, albeit medically qualified experts). I'm sure there are exceptions. My point is that a book whose purpose is teaching (and, at one extreme, exam cramming) is not usually the best source from which to write an encyclopaedia. Would such a book be referred to in a medical review paper? Colin°Talk 19:17, 17 November 2007 (UTC)

I agree with almost everything you've written, and no, undergrad textbooks wouldn't normally be referred to in medical papers. However - at the risk of stating the obvious - we're not trying to produce medical review papers here, we're trying to provide a detailed-but-accessible summary of available medical information.

Undergraduate textbooks have several advantages from this point of view. They're written relatively simply, they rarely include controversial information (which is not to say that we shouldn't include such info, merely that it can be sourced elsewhere), and they are generally excellent summaries of the currently accepted medical / scientific understanding of whatever the subject they deal with is.

Yes, true. But whether that makes them good sources is another matter. I fully agree that reading such books when researching a medical WP article could be very helpful as you describe. We need to explain things in language that is simple and such textbooks are a step towards lay language, whereas books written for experts can be quite opaque. In fact, I also recommend reading government/charity-produced patient-info pages too as they often cut to the essential facts (from a patients POV) and are good at explaining things. The US Gov text is often also public-domain too! I'd still try to find a better source, however. Try looking at autism or coeliac disease and see how much of that is covered in your undergrad textbooks. When you base your text on someone else, it is much easier to do so (without blatant plagiarism) if it is more detailed than what you aim to produce. The act of condensing the text is one way of putting it into your own words. On controversial topics (like autism) it is very helpful to know the ultimate source of the information. If someone challenges your prevalence figures, for example, you can know how up-to-date the survey was, how big it was, what country it was done in, what population group they studied and also search to see who cites that research to find out if it is respected or has been superseded or rejected. It is very difficult to do that with teaching material, that might just give a figure (because, for a student, that is all that is important). Colin°Talk 10:59, 18 November 2007 (UTC)

I'd also take issue with the idea that textbooks are rarely written by a specialist in that field. To take three textbooks I myself have used and know are commonly recommended: Obstetrics and Gynaecology is by Lawrence Impey, MRCOG and consultant obstetrician at the John Radcliffe. Neuroanatomy is by Crossman (prof in Anatomy at Manchester) and Neary (professor of neurology). Even the crap-sounding Cardiovascular system at a glance is by Aaronson (reader in pharmacology at GKT/KCL), Ward (professor of respiratory cell physiology at GKT/KCL) and Wiener (professor of medicine and physiology at Johns Hopkins). I'm not sure these are exceptions - do other contributors have views here?

You've got me there. I'm not medically trained. In other fields, the teachers are good teachers, not necessarily good practitioners or researchers. However, I would argue that a book on a big subject, like neurology, written by one or two authors, is going to have chapters where the author is relatively non-expert. One can't be an expert in all of epilepsy, Parkinson's, autism, brain tumours, strokes, .... Even if the book has several hundred pages, it probably only has about a dozen on epilepsy, which effectively shrinks it to a booklet as as source for that article. Colin°Talk 10:59, 18 November 2007 (UTC)

I agree, however, that exam-question books are not suitable sources, and nor are cramming books - but I think we would lose out by excluding textbooks. Put it this way - people will still be using newspaper articles in medical articles here, and those are far worse secondary sources than medical textbooks... Nmg20 (talk) 23:36, 17 November 2007 (UTC)

Very much agree. This "guideline" needs more work and expansion. I don't think there is a black and white rule or a cut-off point. There is a spectrum of sources and editors should use material as high up the spectrum as they can. Textbooks are better than newspapers, no doubt. Not everyone has access to a university library, and any source is better than no source. Colin°Talk 10:59, 18 November 2007 (UTC)

I seriously doubt that *any* undergraduate textbook covers Tourette syndrome accurately; I would not want to weaken this guildeine to allow their inclusion. SandyGeorgia (Talk) 16:29, 18 November 2007 (UTC)

I think Colin's "hierarchy" model is a good one. There may certainly be some usable info on Tourette's in an undergrad-level text that we wouldn't want to definitively outlaw; on the other hand, if the undergrad-level text is contradicted by a more specialized medical text, then it's clear which we should emphasize. MastCell Talk 20:55, 18 November 2007 (UTC)
I agree with MastCell here - while undergrad textbooks might not cover Tourette's to your satisfaction, SandyGeorgia, I'm betting they'd cover it a hell of a lot better than The Sun/(insert your national rag of choice here)! I agree with Colin's hierarchy too. Nmg20 (talk) 15:03, 29 April 2008 (UTC)
I think we all agree on what WP:V regards as acceptable wrt medical sourcing. However, the change I tried to make here was to clarify what made "an ideal source" or an "excellent secondary source". I'm afraid that undergraduate textbooks are neither, though they still satisfy WP:V. So I'd like to redo that change, if you don't mind, to emphasise what editors on this project consider the best possible sources. In summary: undergraduate textbooks are allowed by WP:V as sources, but our guidelines prefer peer-reviewed review papers and scholarly monographs. Any objections Colin°Talk 18:40, 29 April 2008 (UTC)
Is there such a thing as an "undergraduate" textbook? If there is, it should be secondary to "non-undergraduate" textbooks as IMHO content aimed at students is simplified to ease understanding, and often does not mention major controversies. I generally find recent review articles in high-profile sources much more useful and detailed than textbooks (especially in fast-moving fields where the textbook is easily outdated). Textbooks also have a habit of being low on useful references. JFW | T@lk 05:37, 30 April 2008 (UTC)
By "undergraduate textbook", I mean books bought and read while studying to be qualified in a discipline (medicine). But a general definition of any textbook is a book aimed at a student (whether at school or postgraduate). By monograph (which are usually multi-author these days), I mean a book that a consultant would have on his or her shelf (written by experts for experts). Do you have a different definition or usage of those words? Colin°Talk 07:55, 30 April 2008 (UTC)
No, I think we agree on the definitions. But my point was that undergraduate textbooks still eliminate important content because it has little bearing on the general knowledge required of a medical student, while being highly relevant when discussing a condition in more detail. Monographs can be just as outdated as textbooks, but to me seem to be more detailed. JFW | T@lk 11:36, 30 April 2008 (UTC)
To my mind, an "undergraduate textbook" is one with something in the title which makes clear that it's for undergraduate (pre-clinical or clinical) medical students. It is not any textbook used by undergraduates, some of which are detailed, in-depth discussions of topics which are directly relevant to medical theory but not always practice.
By way of example, I'd be surprised if many consultants have a copy of Ganong (ISBN 978-0071440400) or Kumar & Clark (ISBN 978-0702027635) on their bookshelves, but both to my mind would make first-rate sources, and are liable to be better updated and more widely read than most monographs. That's not to undermine the importance of monographs, merely to say that they tend to be more specific and/or more clinical, and so will not be suitable as references for all parts of an article.
This is probably a long way round of saying "I agree", but if what you're saying is we shouldn't be allowing the Crash Course books (e.g. ISBN 978-0723433507), I agree; if what you're saying is we shouldn't allow Ganong or Kumar & Clark, I don't! Nmg20 (talk) 13:22, 1 May 2008 (UTC)
Ok we agree. But for a given medical article, there's probably only a page or two that is relevant out of the 1000+ pages in those textbooks, and from that you might be able to use it to source a sentence or two on basic body/medicine facts. To write the meat of an article, you need a much more specific reference. Lots of them. That's why I didn't want to recommend them, but neither do I want to exclude them. Colin°Talk 17:11, 1 May 2008 (UTC)