Wikipedia talk:Scientific peer review

From Wikipedia, the free encyclopedia

Contents

[edit] Archive 2

Sholto Maud and Limegreen, with your permission, I would like to archive everything from the top to this section. Sholto Maud, if you would like to continue your methodological discussions in a separate subpage of this talk page, please reply here, and I will archive these concepts, including the concept of the automated parsers for SPR, to a subpage with a name of your choice. We can then continue the thread there. -- regards, --Ancheta Wis 22:49, 20 March 2006 (UTC)

No problem here.--Limegreen 23:01, 20 March 2006 (UTC)
I agree. I have decided to not respond to Shoto Maud above. Archive down to bottom of this section, I say. --Bduke 23:17, 20 March 2006 (UTC)
Agree. Suggest a subpage named "/SPR methodology" for those who are interested in formalising SPR method, and possibility of SPRbot. Sholto Maud 23:39, 20 March 2006 (UTC)


[edit] Reinventing the wheel, authority, and end-of-line.....

Hummmmm, first of all, we do not have to reinvent Scientific Peer Review, and I am sure we should not try to invent a better system. The system has its flaws, but it works good enough for most journals, etc. I have the impression that people make it much harder than that it is.

A author of an article does not peer review his/her own article. It should not be different here. Many articles are specialist articles, and we just might not be able to find someone among the wikipedia editors who can review. Well, though luck. I do not think we should start with externa reviewers immediatly, but that might be a way out. And about all the issues with those, you can ask, they can say no. Professional reviews are secret, so linking it to a version is not an issue. As long as the reviews remain available unaltered, people can always use them again. --KimvdLinde 16:26, 19 March 2006 (UTC)

You must remember that Wikipedia is not a scientific journal. Karol 17:21, 19 March 2006 (UTC)
But still has the goal of being at the front end.KimvdLinde 17:36, 19 March 2006 (UTC)

The problem here is that, even if we are merely an encyclopedia, there are groups who want to manipulate WP to promote hucksterism or patently wrong pseudoscience -- we've all seen examples. These people not only want to have their ideas presented, they want them presented as physical truth. Although WP is not a scientific journal, it faces many of the same problems of authentication that a scientific journal would face. External reviewers could indeed be a good way out.

It's important to remember that external reviewers are not paid and therefore are not part of the value-added that regular journals charge for. In fact, within the scientific community there has been a good deal of tooth-gnashing over the question of what, exactly, the expensive journals are doing for the money we give them.

An issue I have not seen addressed is the problem of editorship. Scientific journals generally have 1-5 editors whose job, in part, is to know pretty much everyone in the particular subspecialty of the journal, and to farm out reviews appropriately: the process is neither self-selecting nor random. Editors in general are overworked, underpaid, and underappreciated -- if we are going to improve/adapt the peer review process, that is the place to innovate. zowie 17:55, 19 March 2006 (UTC)

OK, this returns back to the very earliest discussions. Most of those from the academic community are comfortable with such a structure. The problem is then one of scale: 1) should the board limit itself to natural sciences, because otherwise the task would be overwhelming?, and 2) should the board limit itself to only basic articles, since otherwise the task would be overwhelming? For example, just to help keep perspective: there are 13 thousand math articles alone. There may be similar numbers of physics and chemistry articles. Biology numbers are probably higher. No single board can hope to have mastery over this. The need for scalability in the process is real. Two scalable approaches were proposed. One is a "journal system", where "journals" (small bands of experts) aligned with WikiProjects, are recognized and granted the authority to place imprimaturs on reviewed pages. The other approach is to create a "science noticeboard", similar to existing and highly successful WP noticeboards, where requests for review are posted, and then handled by all comers interested in participating. Either/both systems would use a template similar to Wikipedia:Scientific peer review/Science to record the results of the review. linas 18:09, 19 March 2006 (UTC)
Hmmm. Linas, we may be talking through each other on this one. I believe that the reason scientific journals work at all is that there are a small number of editors (all of whom know and trust each other well) for each one -- the system doesn't scale well. If I understand right, large journals such as JGR often divide the editorship task along very strict subject lines, in order to have a small enough editor pool to handle the papers in each subtopic. The editorial board we've been discussing doesn't have the characteristic smallness, and (as you point out) can't even hope to -- so some other system will have to be tried if a scientific peer review modeled on journals is to be adopted by WP.
Academia solves the scaling problem by having lots and lots of journals, each with small editorial boards. This is indeed one of the two proposals. linas 19:15, 19 March 2006 (UTC)
Although I've put my name on the list of volunteers to help peer review scientific articles, it seems that proper peer review within WP is an ongoing commitment rather than an atomic process as in a journal. Journal articles that are accepted are subjected to a vetting process once (though it may go back and forth a couple of times) and then set in stone: the authors are on record as having made a particular report/argument/statement at a particular time, and the argument is guaranteed to have gone through at least rudimentary checking via an adversarial review. But a WP article can be modified at any time, and even something so minor as changing the location of commas within a crucial statement may change a sound argument into crap that is wrong or not even wrong. No body or scientific review board should put their imprimatur on something that might be changed out from under it! The featured-article system seems to work mainly because so many people are involved in featuring most of them -- so the articles do get the ongoing scrutiny they need, simply because they are on a great many watch lists. But more fringy articles routinely get way out of whack before anyone notices. zowie 18:45, 19 March 2006 (UTC)
Agreed. This is a major concern. I do not see any good solution at this time. linas 19:15, 19 March 2006 (UTC)

[edit] Close nominations

BTW, I think we should halt the nomination process until there is consensus about what the function of the board will be. In particular, I dislike the fact that many of the nominees have not bothered to participate in the discussion here. (or, for that matter, perform the "test review" on science.) What I get out of this is that there are people craving the recognition but are unwilling to do the work. linas 18:18, 19 March 2006 (UTC)

Agree. HereToHelp 18:21, 19 March 2006 (UTC)
Hmmm... At least in my case, my free time comes in small, unpredictable chunks. It's hard to tell whether it will take a matter of minutes or a matter of weeks for me to get to something, especially if it is important but not urgent. Perhaps others are in the same situation. (Though I agree that closing nominations for now is/was a good idea). zowie 19:09, 19 March 2006 (UTC)
I support closing the nomination process, although I would have prefered individual nominees to do it as I did and urged others to do. While understanding zowie's point of view, I too was worried that few of the nominees were contributing. I did not perform the "test review" for several reasons. One, I did not have the time and after studying the article in detail I thought it would need quite a lot of time. Two, I thought it should go to WP:PR. Three, I did not think it was a good test, as everybody could contribute. I wanted a test of getting a reviewer for an article where doing so was difficult. --Bduke 20:43, 19 March 2006 (UTC)

I disabled nominations as best I could on the project page. The markup made this hard to do. I also added a description of the two working proposals, and struck out the failed proposal. linas 19:31, 19 March 2006 (UTC)

The markup isn't that important until we know exactly what we're doing.--HereToHelp 19:34, 19 March 2006 (UTC)
Did you mean to strike out all the nominations? William M. Connolley 19:44, 19 March 2006 (UTC)
Yes. linas 20:20, 19 March 2006 (UTC)

[edit] Two working proposals

The two proposals ignore the proposal that I made, yet nobody has said they disagree with it. The two key features both affect the "noticboard proposal". One, I believe it is vital that we use WP:PR as much as we possibly can. In other words, we use their system and put reviews on their page, as long as we can. Two, I see a role for the WikiProjects in science areas for the "noticeboard proposal", not just the "academic editorial board" proposal.

I dislike the latter proposal. It will be seen as "experts" coming down from on high to fix articles written by "ordinary wikipedians". It will be seen as introducing a SPOV (Scientific point of view), not a NPOV. It will be criticised and might make articles subject to delibrate changes to "muck up" the peer reviewed version.

"Scalability" is really a question of how many people get attracted to helping on review and on how much time they have. It is not really affected much by process. "Authority" is a dream. This is a wiki. All we can really expect is that a peer reviewed version will have lots of editors watching it on their watch list; that they will correct wrong changes; and that the authority of the peer review itself will be a good reason to explain to the editors who have made changes why they should not have made the change. We must let the magic of the wiki work, because we have no real alternative. Of course, the WP 1.0 project, or printing a version as the Germans are doing, may provide both a scope for peer review and a way of sealing in the authority of the peer reviewed version. Neither is there in the standard WP, so it is no good hoping they are or can be put there. --Bduke 21:07, 19 March 2006 (UTC)

When I signed up, what I was signing up to was a simple "review" of articles, essentially for scientific respectability: statements properly referenced to reasonable sources, mostly. Since that time things seem to have got rather confusing. William M. Connolley 22:10, 19 March 2006 (UTC).
Indeed. And there's no light at the end of the tunnel visible yet.--HereToHelp 23:43, 19 March 2006 (UTC)

[edit] Scalability, authority.

I personally do not think scalability is an issue. Bluntly, most articles are not worth to be peer reviewed. They are either written poorly, incomplete, ununderstandable etc. If we want a peer review system to replace WP:RfC, and the related stuff, it is not going to work.

Authority. You can have the number one authority in the field doing areview, and it can be biased as hell. Just because he does not like it. That is why you generally have two reviewers. And that is why you can protest. The question is this. Do we want NO peer review, or are we going to take the best we can get for the moment, although it might be not the best we could get? The latter will give us at least somewhat more than hat we have now.

Yes, texts are going to be changed, and that is one of the major powers of wikipedia. Without perr review, and by generally non-specialist, we are close to the same level as the Brittanica, build by specialists. We can do better, and peer review can make a contribution. I see peer review more as the following. A group of editors has worked their butts off to make a good article. They are good willing, but realise they are not the experts that they would like to be. The article is getting to a point they know it is good, but would like to see what an independent expert thinks of that. That is the type of articles that is most likely to be asked for peer review, and that type would be the ones I would be aiming at. As soon as the peer review is done, new insights, additional information etc is going to be added. And yes, articles can go bad because of various reasons, including POV pushing. Though luck. Revert. That is why I have 300+ pages on my growing watchlist. That is part of wikipedia. In my field, there is a core article,. that is so bad, I was amazed that it got accepted. I know one of the reviewers, and he had recommened that it was fully rejected. That reviewer now uses it in classes as an example on NOT to do that type of research. If we want 100% correctness, from 100% authorative reviewers etc etc etc, we are not going anywhere. I am more in favour of lets start, let the wiki system work. In the coming year, we will probably change things and over time, it will get a better shape. And if it does not work, we will know. KimvdLinde 00:35, 20 March 2006 (UTC)

OK. I was trying to describe what I thought would be "the happy middle ground" and instead of pleasing everybody, I see that I pleased nobody. Bduke and KimvdLinde seem to be saying that its all hopeless, that there is nothing that can be done, and that this effort should be ended -- this is a complete about-face from what I saw a few days ago, when people still seemed eager and hopeful. I'm going to withdraw from the conversation for a while, since this all seems to be turning toxic. linas 05:50, 20 March 2006 (UTC)
Sorry, maybe I should not say some things sometimes.... I am definatly saying it is NOT hopeless. I see very good opportunities, and I think it is very important. I do however not see some of the bears others see. If me saying that we can not achieve 100% correctness, from 100% authorative reviewers etc etc etc, is saying it is hopeless, I am equally saying that real world peer review is hopeless, which I do not agree with. Ideal, no, usefull, yes. KimvdLinde 14:51, 20 March 2006 (UTC)

I have not said it should all be ended. Would you please comment on what I proposed at #Another proposal? Nobody has commented on that. I have said that the current proposal is dead. We will never agree on a Board at this time. I have also said that we should not move away from wiki norms any further than we have to. --Bduke 06:06, 20 March 2006 (UTC)

The spirit behind this project is a good one. What the end process will be I do not know. Here's my take (having only scanned about 3/4 of the discussion thus far). I like the "requests-for-scientific-review" bulletin proposed above. Perhaps from that could grow a more formal peer review process. I would watch it, though I would not join a board. Kudos is nice, but if I wanted recognition I wouldn't be on WP, I'd be publishing. Regarding what is an "expert", I'd say anybody with at least a PhD counts (for their field). That said, I'll get my PhD in a matter of months, yet I read a whole lot of rubbish in the hydrology and geomorphology pages. I have published, presented, reviewed, and acquired funding; I'd say I'm an expert (in my field). I already monitor the areas in my fields; I'm more inclined to fix them up as a normal WPian than via peer review. As for the nice Nature review, I don't really care. WP is trying to be right, not merely better. EnBrit has professionals, but are they "experts"? Establishing a scientific community on WP would be great. More power to us. Cheers, Daniel Collins 19:59, 20 March 2006 (UTC).

[edit] Re-visiting an idea...

I'm going to move Bduke's post all the way down here so that we can take a look at it - nobody has responded to it probably because the idea has sort of gotten lost amoung all the posts I think. It was done with this edit: [1] --HappyCamper 12:33, 20 March 2006 (UTC)

I don't understand how Bduke's "proposal", above and below, differs from how things have been done in the past. linas 14:34, 20 March 2006 (UTC)
I don't really know either...maybe that this page simply focuses more on "science related" articles? Maybe we wait for Bduke to respond. --HappyCamper 20:55, 20 March 2006 (UTC)

[edit] Another proposal

Spending a lot of time away from my terminal often improves my thinking and I have certainly been away from it for most of the weekend. Much as I think a Board could be usefull and will probably be needed at some stage, I think we should avoid it now. There is simply no consensus about whether we have one, what it would do, and how big it should be. The reasons are complex and I am not sure I can properly articulate them clearly. Part of the reason is the distrust of experts on Wikipedia or more precisely the distrust of experts who appear to be more important than other editors. I think we should stop discussing whether expert reviewers will get credit for it in their institutions or whether the article version that includes the results of their review should be labelled in some way. We have to accept the wiki way. If experts will not review then I'm afraid there is nothing we can do about it. We must "let the Magic of the Wiki take over" as HappyCamper said above. We must also fit into the normal processes as much as possible. So my proposal is:-

  1. Use the normal Wikipedia:Peer review. Identify an article, tag the talk page with their tag and put the review comments on their review pages. WP:PR may have its problems but I do not think it is broken. We should try to make it work for us. The trick now is to get experts to add to that review process.
  2. Also tag the talk page with, for example, the {{subst:expertBiology}} tag as well as the WP:PR tag. We would use something very much like Wikipedia:Peer reviewers and the person adding the tags would use that to seek a reviewer.
  3. Let us make use of the WikiProjects such as Wikipedia:WikiProject Chemistry. At the beginning of this WP:SPR I put the idea to the talk page of that project. At least four participants in that project have contributed to the discussion here. Have the other science WikiProjects been consulted and asked for their views? Some one in each project could list the new entries in the category attached to our tag on their WikiProject page. This might also attract expert reviews.
I know Wikipedia:WikiProject Physics was notified. Karol 08:24, 19 March 2006 (UTC)

So, I am sorry Adam I think we have to mothball this idea and have a different page that explains what we are doing. I just do not see it working. I think we have to try something much simpler and much more in the Wikipedia spirit. After a few months we can review it and see where we take.

The only snag I see is that we might want to tag articles for review that are not close to featured status. If people on the WP:PR talk page do not like this and say remove the tags, we try to convince them. If we can not then we will have to put the review on a page we construct, but let us wait until that is forced upon us. --Bduke 08:07, 19 March 2006 (UTC)

I'd just like to point out that this is the first really intense Wikipedia discussion on this subject, at least out of what I've seen. Karol 08:24, 19 March 2006 (UTC)

I think so too. When this project takes off, WP:SIGN will be notified, and we can credit Adam for being instrumental in starting up with the idea. --HappyCamper 12:37, 20 March 2006 (UTC)

[edit] Comments

I like Bduke's idea. Why not leverage the existing Wikipedia:Peer review system? It's not perfect, but it works. I did not want to say this too early, but projects like this take months to set up, and at least this project has been able to get the attention of lots of people. So, it is at minimum, on the mind of a few people already.

Bduke, actually, your post has raised a number of issues that I'd like to address, but probably best not to go too deeply into them because it might precariously sidetrack everything. I will say that the editing environment on Wikipedia is different from most places in academia - we need to target this project to address that environment, and the needs in that environment.

I like to think of it this way - and I am deliberately writing this with the perspective of "unveresity/collge" in mind - I also have similar analogies for those who work dedicatedly in industry et cetera...but we will focus on this particular perspective for a moment.

Think of creating articles on Wikipedia as creating course material for your students. The result of what we do here should be that you would feel comfortable giving the article to your students as introductory material to a particular topic. You would proudly show off these pages in class and in your lectures - and use them with confidence. Partly because you read through it the week before your lectures, but you notice that you didn't have to put so much effort into preparing your notes for that day...hmm...I wonder why? :-)

The article should contain information for everyone - targeted to smart, intelligent people who want to learn, and who have the drive to find the information they need. Wikipedia has a host of articles that are obviously on graduate-level and post-graduate level. If I may take some inspiration from the current peer review system on Wikipedia, most topics being reviewed are rather common things known to everyone, but just that no-one has bothered to articulate all the intricacies of the topic and put everything together. Take blank page as an example. Lovely article - and I bet that it would be pretty hard to find any better information on the internet about it in such a quick way. Hence, let use frame whatever peer review project we set up, as targeting primarily undergraduate education; something beyond the typical high school, but not quite so cutting edge that reviewing the article would benefit very few of the masses who visit here. This is a reasonable compromise between all the possibilities out there, and very much worth the consideration.

The discussions on Wikipedia are not anti-elitist per se - imagine it being more like this - you are in a room with perhaps 10-20 people. It consists of faculty members, undergraduate students, university officials - all in the same room, all with different perspectives. All are trying to figure out how to improve the current education that the undergraduates are receiving. On a talk page on Wikipedia, the traditional barriers which prevent good feedback are not there - people who engage in genuine discussions are those who care. We simply need to remember this in the back of our minds - and in doing so, we avoid all the pitfalls that mistakenly give the perception that Wikipedia is hostile to experts - it is not. It is simply different, where the expertise of someone is completely exposed to criticism, misunderstanding, or whatnot. This is the negative side of it. Knowing this, the expert is also keen to know how to work the system to their benefit - bring out the positive side of editing here. Pick the arguments that are worth fighting for, where there are people willing to listen, and where it will make an impact. What we are doing here is precisely this.

In my opinion, this project simply will not even need to consider contentions issues related to pseudoscience or whatnot. The fact, is that we will not offer any useful feedback that would be healthy for Wikipedia - addressing these articles is simply for another group on Wikipedia to worry about - the editors of those articles know that too. We don't need to worry on their behalf. What we can do, is be welcoming, and do the best we can if topics like this come up for review.

Now, I wrote this with some things in mind...people are leaving this talk page out of frustration. This is a project that will take months to set up. But it's all worth it. Simple is best - work incrementally, and gradually usher in ideas and changes that will benefit the project. The Wiki approach does work, but it just needs to be given time to mature - that is, for those active here to get a feel for what other people are like on this talk page.

Alright, I really, really need to head off...apologies for unrefined posts and streams of consciousness. Closing points:

  1. Target undergraudate education
  2. Don't worry about psuedoscience - that's for another group to worry about
  3. Focus, focus, focus.

Bye!! (And rushing off...) --HappyCamper 13:17, 20 March 2006 (UTC)

There's many different undergraduates, but an article can be tailored to have parts for the 1st year students and parts for the senior majoring in the subject. Starting from a WP article, it's relative easy for anyone who knows the subject to add more advanced topics. We need some way of adding a meaningful elementary explanation to pages such as the mathematics section, and , just as Happy Camper says, what's needed for this is not necessarily great scholarship, but great skill in teaching. We all know excellent researchers who present excellent beginning courses, and are proud of it. (I do not mean to imply that I am one, )

Some of those currently working on the math pages pages have deliberately written them to be at the very-bright-senor-undergraduate-major. One thing we could usefuly do with peer review is to encourage them to write otherwise DGG 22:59, 24 September 2006 (UTC).

My idea was indeed to lever in SPR starting simple. How does it differ from the status quo?
  1. Our "expert in physics", or wahtever, tag is added to the talk page of the article as well as the WP:PR tag.
  2. This might add the article to "Category:Articles in Physics needing review".
  3. The article is listed on our page under subject categories.
  4. We try to build up small teams in each of the Science WikiProjects to draw attention to the article on their page and seek reviewers.
  5. This is a kind of building a de facto board for each specialism. We let it grow naturally.

How does it differ from other proposals including Adam's original proposal?

  1. There is no formal board.
  2. The review report does in the current WP:PR place (at least as long as that is possible), not on WP:SPR.

Does it deal with the issue of "authority"? Not directly. I think the wiki way will work here. The fact of the review will be on the article's talk page. The date and time when the review material was incorporated into the article will be there. The page will be watched. The presence of the review is a powerfull argument for reverting silly additions.

Does it deal with the issue of "scalability". Yes, as much as any other suggestion does. It all depends on how many people get involved. We can not predict this or manage it.

I think I have covered the essentials of my proposal, perhaps clarifying it in part, but not adding anything new.

Other points:-

  • I fully agree that we should have articles in science that do not go beyond the undergraduate level and that those that do go to that level would be important teaching resources. I think we have some that do go beyond undergraduate level. I'm afraid, HappyCamper, that the article we have both contributed to on Couple Cluster theory is one of them. We should simplify it and clarify it down to what an undergraduate might want to know or for someone in a different field of chemistry to just get an over view. There must be many articles like this.
  • This point is so important that we should put it on the WikiProject pages as an important criteria to bear in mind when writing or editing science articles. However, let us get consensus here first and have a proper draft of what we might say on the project pages. HappyCamper, I think you have made a simple but brilliant contribution here.
  • I fully agree about pseudoscience.
  • I hope I have not being pushing people away in frustration. I'm going to limit what I say on this page for a while. --Bduke 22:08, 20 March 2006 (UTC)
    1. Support. Let me know what I can do to help. The templates are one possible avenue for me to aid the project. --Ancheta Wis 22:42, 20 March 2006 (UTC)
I'll support that. Like regular PR, but with specialists commenting? Sure. Just leave room for the non-experts to comment to (separately).--HereToHelp 22:49, 20 March 2006 (UTC)

[edit] Britannica 's review of Nature's review of EB vs Wikipedia

There is some good scientific reviewer/reviewee interplay in Britannica's review of Nature's review, Britannica vs Wikipedia. --Ancheta Wis 09:28, 23 March 2006 (UTC)

I read so many science and nature articles, that it does not realy strike me that they publish bad science with wrong facts additionaly leaving away some needed facts. This is nature at its normal standart.--Stone 09:37, 23 March 2006 (UTC)
Re-reading, EB is standing by single reviewers. This highlights a potential advantage to having multiple reviewers for SPR. --Ancheta Wis 09:54, 23 March 2006 (UTC)
I knew Nature is quite careless, but I'm surprised they could be so careless. Well, the fact that WP was compared to EB is alone already a success. Karol 10:07, 23 March 2006 (UTC)
Britannica's response (and their public comments on WP in general) was pretty predictable and entertaining. It is clear they are scared of WP based on how publicly they are willing to denigrate it. But even with that in mind, it's pretty clear to see Nature did some shoddy journalism. I'm not sure why they bother with the journalism type articles if they're not going to do them more carefully than this. This is also why I was pushing for people to not just change the articles in the Nature review to reflect what they said, but check it to multiple sources. But I agree with Ancheta, this points to the importance of multiple qualified reviewers. - Taxman Talk 20:08, 23 March 2006 (UTC)
The fact that Nature's reviewers have mistakenly considered true statements to be false is inevitable and uninteresting. However EB needs to demonstrate that these mistakes favor WP, that is, that it happened less often in the case of WP articles. They have not made any mention of such errors on WP articles, which may mean that they also investigated errors on WP articles and found that Nature made a similar number of errors.
Of course arguing about 30% of the errors misses the point. The substantive difference is that EB is better written and more coherent, whereas WP contains more information, is more up-to-date, has links that may be followed instantaneously to its sources and to additional information, and is free. JarahE 19:58, 23 March 2006 (UTC)
The other thing is that Wikipedia contains more obscure information, particularly about scientific subjects. For example, in the article about Dirac, Brittanica makes the case that a lot of the omitted information isn't suitable for a general encyclopedia article. In Wikipedia, the idea seems to be that things like Dirac's work on magnetic monopoles and his law of large numbers, as well as numerous other things, merit inclusion because Wikipedia is not a paper encyclopedia and can contain arbitrarily detailed technical information. –Joke 21:03, 23 March 2006 (UTC)

[edit] Where now?

There has been no comment on this project now for nearly a week, except for the Britannica discussion above. What does this mean? Does it mean that I have driven everyone away, or does it mean that people want to proceed with the rather minimum proposal I put forward. If the former, just tell me and I'll apologise and go away. If it is the latter, and there has been some expressions of support, where to now? I would prefer it if someone other than myself, put the plan I proposed on to the main page. I think this would make it clear that we are all agreed about the details of the plan. If it is on the main page (and I suggest all former proposals removed), we can then debate it further to refine it and then hopefuly get some peer review actually done. What do people think? --Bduke 23:52, 26 March 2006 (UTC)

I think people just wanted a little coffee break from this discussion - as for me, I've just been a bit busy lately. --HappyCamper 23:59, 26 March 2006 (UTC)
On the undergraduate level thing, I agree with Joke not with Bduke that, for me, the reason that I find Wikipedia to be useful and EB to be useless is that Wikipedia contains fringe topics. EB's model of encyclopedia building is not scalable in the sense that it cannot contain advanced material, it would be too expensive to write/assemble/edit it all. EB, as a result, to my knowledge has never been useful for scientists, although it may be useful for undergrads.
The archive, on the other hand, contains pretty much all of the advanced material in existence, is very useful and has revolutionized the way we do research. However it's very poorly organized, you more or less need to know what paper you're looking for in order to find it. If you want to know something outside of your specialty, its useless.
WP has the potential to fill this gap perfectly. It is organized by subject, not by some irrelevant numbers and difficult-to-guess keywords, so it is easy to search. It doesn't need to contain as much technical information as the archive, because it has links. In fact, due to the no original research policy, all original research is referenced via links. But really if the pages are at an undergraduate level, they won't always be able to convey enough information to let researchers know which link they want to click.
Fortunately WP is big, and next year it'll be much bigger. So I think there's plenty of room to address the needs of the undergrads and the researchers, sometimes even on the same pages. I'm teaching a mixed grad/undergrad class next month in which I'll suggest that the students look at some WP pages. JarahE 00:23, 27 March 2006 (UTC)
On reflection, I do not disagree that much. There are a lot of pages that contain stuff way beyond undergraduate level and there is no way we can delete them, so the point is moot anyway. However, I think we should have introductions to each article that are unstandable by undergraduates or people who do not need a really detailed knowledge of the topic, with the more advanced stuff later. There is a discussion on the en:WP e-mail list about using the introductory paragraphs of articles to publish a printed concise Wikipedia. Not sure I agree with that proposal, but you can see the point. --Bduke 01:06, 27 March 2006 (UTC)
Ok, so I think we're in agreement. Introductions should be accessable to undergraduates. Each category also has a main page, it'd be good if this were also as introductory as possible and described how the other pages fit in. JarahE 14:08, 27 March 2006 (UTC)
Personally, I think this project has already done enough - got people thinking about SPR in WP. The page itself seems to have "burned out", in my view. The brainstorm lasted... and now things probably need some time to settle in, probably more than a week. Karol 00:27, 27 March 2006 (UTC)
I'm going to leave it for certainly more than a week. --Bduke 01:06, 27 March 2006 (UTC)

This page is still on my watchlist; if anyone wants to try to seriously revive it, I'll help (that's what I'm here for!)--HereToHelp 00:38, 27 March 2006 (UTC)

Certainly has been an interesting brainstoorm. A week later I'm still confused as to whether SPR means 'scientific' peer review, or peer review of scientific articles (PRSA). Either way, in terms of where to now I believe that it would be of benefit for people to consider formally stating an "SPR" method and criteria for review. These questions have motivated me to conduct something of an ongoing review of dicussions of the OR Wikipedia policy. Charles Stewart, made some interesting suggestions early on.
"I'll suggest that we should be seeing at least 7 citations apart from the core propoents, at least 12 articles principally devoted to the topic, and twenty or so general comments on mailing lists." Wikipedia_talk:No_original_research_(archive_1)#Where_to_draw_the_line.3F, 00:02, 16 Sep 2004 (UTC)
Sholto Maud 02:47, 27 March 2006 (UTC)

Please, let us not get confused about what we call this process. It always has been about peer review of the articles in Wikipedia about science, with some debate about how we define science (e.g. do we include social sciences, mathematical sciences, medical sciences, or just stick to natural sciences). Let us also leave aside that debate of defining science. --Bduke 05:17, 27 March 2006 (UTC)

I agree and thank User:Bduke for the many useful contriubtions. Without meaning to create unhelpful controversy, I understand there to be a difference between the notion of "scientific peer review", and that of "peer review of scientific articles". The former implies the notion that the peer review process is in some way "scientific" and therefore that the article quality and reliability might be maintained through a scientfic method which embodies some quantitative, measurable property. On this point, User:Charles Stewart says that the following appear to be measurable:
  • Many researchers beyond the original proponents apply the ideas contained in an article
  • discussions on mailing lists
  • articles in field are widely cited

(User:Charles Stewart, Wikipedia_talk:No_original_research_(archive_1)#Where_to_draw_the_line.3F, 00:02, 16 Sep 2004 (UTC))

However User:Charles Stewart also says "I am not sure what kinds of numbers to attach to them". "Peer review of scientific articles" is more open and does not appear to pressupose any measurable criterion in the peer review of the articles in Wikipedia about science. As a consequence of this view, the peer review process which aims to maintain article quality and reliability is done by some informal or arbitrary method of content and citation evaluation. That's the difference I see: one method is scientific with agreed units of measure for article-quality evaluation, the other is non-scientific with arbitrary measure for article-quality evaluation. I put the case that any SP review board would appear to have to choose between these approaches when answering the question of where to from here. Sholto Maud 08:30, 27 March 2006 (UTC)

OK, while I do not accept your view that we have to have measurable criteria for reviewing scientific articles, here is my take on it. A decent review would in my opinion ask the following questions:-

  1. If the topic has a broad accepted knowlwedge base, is the material an accurate reflection of the scientific consensus on a topic?
  2. If there is still debate about the content of the article, are all views fully covered and expressed in a NPOV way.
  3. Is the article complete, covering all material from science that comes under the heading of the article title, not of course in real detail (complete in itself, or having the material in sub-articles. For example the Post Hartree Fock article does not cover all post HF methods but links to articles that do).

All three of these need an expert in the area to answer. In no case are we measuring something. I suspect that an emphasis on measurement here is counter productive. It might even be the numerical fallacy. That is my take. --Bduke 09:24, 27 March 2006 (UTC)

Sholto is bringing up the fringe-science issue: the increasing appearence of WP pages on topics in the grey area of fringe science/pseudocience, haveing very few followers, and advocated by questionable personalties seemingly having boundless energy to edit. A request for citations is a common mechanism for beating back these articles.
I share JarahE's vision that one day, WP might fill the gap between textbooks and Arxiv. However, in doing so, it will increase the amount of friction between authors who have fringe or even-wrong views on arcane topics. Uranium trioxide and plasma cosmology come to mind as examples of incendiary topics that are hotly debated. linas 21:58, 30 March 2006 (UTC)

[edit] How about this?

Let's try this? Completely off the top of my head. Let's have a brave Wikipedian do a peer review writeup for any article of their choice, post it here, and have it thoroughly scrutinized and surgically analysed so we know exactly what it takes to do a peer review. A trial run will give us an idea of how much time is needed, what sorts of things we should include, what things are missing, et cetera, et cetera.

If nobody volunteers, I'll do this - by the end of May. Of course, anyone is more than welcome to beat me to this initiative...so if it would be a healthy endeavour, this is what I'll try and puruse. Thoughts? --HappyCamper 22:41, 27 March 2006 (UTC)

cool! I think Samsara was attempting somethign like this before leaving us. My wishlist,

  • rigorously log the amount of time it takes you.
  • log the kind of resources (books, articles, databases) you have to consult to verify the article.
  • estimate the quality of the article after first read.

Sholto Maud 03:38, 28 March 2006 (UTC)

OK, HC, I propose to review Computational chemistry. I have edited this article, but only really tinkering. I have been putting off making a major effort and reviewing it will precede a major editing effort. This page is in a sense a portal into the articles that I edit most in my professional expert capacity. It could be a featured article. Following my earlier suggestion, I propose to put it into the main Wikipedia:Peer review process. I will however follow your suggestion and put my review here and then later add it to the normal Wikipedia:Peer review page. I will not add the SPR tag to the article. Whether we use that is I think still to be decided. I'll follow Sholto Mauds suggestions, but I'm not sure what the time tells us. People take different times over a task and different articles will take different times. --Bduke 07:26, 28 March 2006 (UTC)

Sholto Maud reminds me that we should do something about Science, but it can wait for now. --Bduke 07:47, 28 March 2006 (UTC)

Well, I just thought of something completely crazy to go about doing this...why not put in an incomplete review (there is no reason why a single Wikipedian needs to write the whole thing), and let the Wiki process take over? Unless this turns out to be an absolutely poor idea, there's no reason why we can't mark reviews for science and computational chemistry as "incomplete" and simply work on these until we think the review itself looks polished. For SPR, what we do, is we say, put a cap on the number of incomplete reviews - say, at most 3. Then, anyone who's interested in improving these reviews does so, and on their respective talk pages, people decide when it is "done" - done in the sense that it would be useful for whomever is writing the article to come by and make use of the review. No reason why a review needs to be finished - articles on Wikipedia constantly change, and the review and commentary that goes with it can change too. Well, I don't know...we just need to start from somewhere because we don't know what a review from this place is supposed to look like. Right now, we have no idea. I suppose like most articles on Wikipedia, we start from scratch! I'll jump in and do something in a little bit. I really don't have the time to sit and think about this at the moment. But as I said - by the end of May, I hope to have something to share. --HappyCamper 13:50, 28 March 2006 (UTC)
Oops, I missed Sholto's comments - well, if we're daring enough to try writing a SPR on Wikipedia, then everything is logged, and we'll know how much time was spent on it. However, the review would essentially lose its primary authorship, which might not be what is needed here to convey an "authoritative" review. In my opinion, this is not so much of an issue. Trust is something that I don't quite understand on Wikipedia, but I do believe it if someone says they know X, Y or Z - all I need to do is to look at their edits. Those in the know aren't fooled, and it is extremely difficult for someone to pass off as an "expert" on Wikipedia - there is a certain maturity and quality to an expert's edits which is difficult to describe, which distinguish them from say, the "average Wikipedian". I think before we even worry about whether a review is convincing enough for the skeptical public out there, let's convince ourselves first that we can generate results which are meaningful to the project. Do this consistently, and the skeptics are won over. I am wondering whether the effort to write a review would be better spent improving the articles themselves, but then again, there's no point in wondering if one can actually do this and answer it once and for all! As I said, for me, end of May...end of May...end of May... --HappyCamper 14:04, 28 March 2006 (UTC)
I think it's a very good idea (and very much the Wikipedia style!) to just be bold and give it a try! I don't think a "partial review" is a good idea IMHO. I think each reviewer's perspective is different, and one only gets the full benefit from that once a complete review has been done. The collaborative value should come from bringing together several complete reviews, not several incomplete ones. I think, also, that the time taken on a review will depend on the time you have available - if you on a break you may take the time to check every reference, but if you are busy you may not. In addition, a lot depends on your access to the literature - where I used to work in rural Vermont, even looking at a recent JACS involved a 100 mile drive, whereas now I have it electronically on my desktop, full text. However, it's certainly reasonable to keep track of time spent on this task. Walkerma 05:35, 30 March 2006 (UTC)

[edit] Re: Time log

What isn't logged is the amount of time that is spent in libraries finding books/articles, and then reading them in order to verify an article.

Re: usefullness of logging time, if we get an accurate reading for the amount of time embodied in construction and SPR of an article, we might then approach an estimation of the monetary value of the article, which is one (perhaps not the best) way of measuring the value of the article. Let's say SPR is worth about €30 per hour. Let's say normal article editing is worth about €10 per hour. For example, if the computational chemsity took 20 hours of normal editing and 10 hours SPR then the monetary value would be (€10 * 20) + (€30 * 10) = €50.

I'm sure that there are many other stats that can found from this kind of measure. Sholto Maud 21:32, 29 March 2006 (UTC)

Oh, I see. Okay, I'll keep that in mind then. --HappyCamper 21:48, 29 March 2006 (UTC)
I don't think that there exists any meaningful estimate of the value of a review. You could ask what it's worth to the reader, but the reader probably wouldn't pay for it if asked, although he gets value from it. So do you say zero? Or try to guess the value he gets? You could also ask how much the reviewer would get reviewing for a journal instead. Personally I've reviewed for three journals and I've never been paid, I don't know if more famous people get paid, but I'd guess it's rare. In general, reviewing pays for itself in that you learn by reviewing, plus you can put it on your CV and you hope that it'll be appreciated by the editors, who tend to be important people. So should these benefits be subtracted from the cost? In my opinion there are a lot of questions here, and there are no answers, other than that the value of a review is ill-defined. But I can say that reviewing an article takes about one afternoon, at least if its in your specialty. JarahE 22:14, 29 March 2006 (UTC)
That's true - frankly, it wouldn't be too meaningful for me either, but I'd keep track of how much time I spent on it anyway...It's just something fun to do along the side. --HappyCamper 00:59, 30 March 2006 (UTC)

[edit] when does the discussion end?

If anybody can give me a note if the first article is to review. I will unwatch the SPR pages, because this discussion leads everywhere without knowing what the actual task will look like. I am an experimatal scientist, who wants the system running. If something is wrong a discussion can be done, but to do several pages of philosphy what the SPR will look like is not aware of how wikipedia will work with this project. The wikies will use the project in the way they will use it and to speculate about if an idiot wants an SPR about spoo or physics in alternativ universis will come up or not even if it is discussed bevor. --Stone 08:04, 30 March 2006 (UTC)

Current nominations. Status TBD:
  1. Science review by User:Samsara; comment by User:Linas
    My suggestion on this one is that we put it to bed. I suggest we aim to edit the article to reflect the comments from Samsara and Linas. It was an interesting learning process. --Bduke 17:59, 30 March 2006 (UTC)
  2. Computational chemistry * I will do this one. But not now. --HappyCamper 13:56, 30 March 2006 (UTC)
    When I suggested this one, I said I would do it. Delighted to have your contributions too, HC. I planned to start it next week. I was waiting in part for comment on my proposal which was to trial it through the normal WP:PR process, with our comments here in WP:SPR but added to the WP:PR page at some stage. Nobody seems to have commented on this. --Bduke 17:59, 30 March 2006 (UTC)
Put something on the talk of computational chemistry page!--Stone 06:31, 5 April 2006 (UTC)
  1. Supergravity
    I'm not sure this one is active. I suggested it earlier as an example of the Board searching for a suitable reviewer, but it then became clear (at least to me) that we would not agree on a Board, so now we wait for people to offer to review an article. Nobody has offered on this one. --Bduke 17:59, 30 March 2006 (UTC)
Supergravity was meant to be a straw man, but was never knocked down. It was meant to be an example of a WP article for which not only does the current WP process works just fine, but also can only be reviewed by a rareified few. But I'm rehashing old stuff. p.s. User:JarahE, please add your name to Wikipedia:WikiProject Physics/Participants linas 22:10, 30 March 2006 (UTC)
Done (ummm, I added myself, I didn't review the SUGRA article, which really I think still needs more work before it fits the stable criterion anyway, the last subsections are still blank).JarahE 17:27, 5 April 2006 (UTC)
As for knocking it down, I readily admit that it's not well written, but except for the recently added anonymous remark on SU(2) lying within the Poincare group I stand by everything on the page and am confident that a reviewer will not find factual errors. It's a shame that red reviewers haven't been invited to review, really in my limited experience I've found that some of the best informed edits are made by anonymous IP addresses, there's a difference between a good Wikipedian and someone who can spot factual errors.
Anyway, voting may not do much good yet as it seems like there was only one nominee per field of study, but Wikipedia is growing fast and many of the current nominees did seem qualified, so I think its a shame that the peer review project is abandoned because no meaningful voting is possible. JarahE 18:42, 31 March 2006 (UTC)
JarahE, please don't feel you're not part of the community because you have a red linked username. I'm sorry if you do feel left out because you haven't been invited to review anything. It's partly a cultural thing on Wikipedia: Be Bold!. Long time contributors are simply used to jumping in and doing whatever they think will improve the site - sometimes, we forget that it's always nice to extend an invitation out to others to be bold too. So, with that, if you like, feel free to review anything you please and share it! --HappyCamper 17:59, 5 April 2006 (UTC)
Thanks. Really, I'd never imagined that I'd face racism because of the color of my name, it's kind of nice, feels futuristic. JarahE 19:42, 5 April 2006 (UTC)
A red username is usually—usually—the sign of a new user who may make edits that need to be revised. It's by no means a perfect or fair system. I think you should just create a page, even if it's only one sentance, and be done with it.--HereToHelp 20:21, 5 April 2006 (UTC)
You can create a userpage and keep the red link, if you just made your signature red in your preferences. Best of both worlds :-) Wikipolitics and wikihistory aside, more often than not, I see newcomers needlessly bitten, and this is an unfortunate thing in the system. --HappyCamper 23:09, 5 April 2006 (UTC)

[edit] Wikipedia:WikiProject Military history/Peer review

Have you had a look at this peer review system? Maybe we should model the WP:SPR on this page. I like this quote from that article:-

  • All reviews are conducted by fellow editors—usually members of the Military history WikiProject. While there is a general intent to expand this process to allow for review by subject experts, the preparations for this are not yet complete.

Like us, they have not quite figured out the "expert" part of the process, and leave it to members of their Project. We have members in a tree of Projects from Wikipedia:WikiProject Science through all the discipline Projects. I am still inclined to also call for the normal peer review as that might attract more people to the process, but we might leave some reviews only on WP:SPR if the articles are not already close to featured article standard. If people agree, I'll rewrite the main WP:SPR page. --Bduke 02:09, 31 March 2006 (UTC)

It's an interesting idea. If we maintain a list of these editors (at least the ones frequently active here) and their credidentials and WikiProject affiliation, we're practically back to where we started. I'll support for two reasons: (1) it's better than nothing (that is, what we have now) and (2) Wikipedia will probably get a massive overhaul and fact check from external sources as time goes on. We can't verify credentials over the internet (and especially a wiki) so we might as well leave it to those who claim to be involved. I'll assume good faith and say that anyone with a well-established account who takes the time to write a review that passes the common sense test is qualified. Some people won't care about what system we use; they still won't trust us. But that doesn't matter. Let's do something to make things better, even if we don't get praised (immediately) for it.
To that end, let's invite everyone who applied and looked decent (no "redlinkers" yet; but students are welcome to review: the more, the better) to review a range of articles. Again, we can't assure utmost quality so let's go f0or quantity. Let's always keep a good handful of articles out to be reviewed so everyone can participate: remember, the astrophysicist wouldn't do as well on a topic like vitreous humor (which has nothing to do with jokes) as (s)he might on string theory. So, summarized: let's do something to get this moving. We're being too picky.--HereToHelp 03:06, 31 March 2006 (UTC)

[edit] Review of Computational chemistry

I have added a page linked from the main SPR page as Wikipedia:Scientific peer review/Computational chemistry and added my review or at least a first draft of it. I have also copied over Stone's review from Talk:Computational chemistry. Please add other reviews and comments on my review. --Bduke 03:13, 7 April 2006 (UTC)

[edit] Cleanup of main SPR page

I have started by adding two pages for active recent reviews and archived old reviews. The former is transcluded into the main page. The latter is transcluded into the former. To edit the former click on "Edit here" above the heading "Scientific Peer Review - Recent requests for review" and edit in a new entry. To edit the later, click on the link titled "Archive of completed reviews" and edit it. The purpose of all this is that the page for active recent reviews can be transcluded into every Science WikiProject page to draw attention to the articles. It probably needs a bit of explanation adding before this is done, but I'm getting there. --Bduke 05:15, 7 April 2006 (UTC)

I have been bold and considerably trimed down the main page to reflect what we appear to be doing. The old stuff is still available from the History if you want to add any or all of it back. I have made use of the {{scipeerreview}} tag and dropped for now the idea of using the normal WP:PR process, although I think we should. I do not like having both the SPR tag and the {{peerreview}} tag on the article's talk page. I have still to add the transcluded bit on to the Science WikiProject pages, but I will add it to one as an experiment and tell you here that I have done it. Please comment. --Bduke 06:32, 7 April 2006 (UTC)

The transcluded list now appears on the Wikipedia:WikiProject Chemistry page. --Bduke 06:48, 7 April 2006 (UTC)

I have implemented the suggestions of Wikipedia:Scientific peer review/Science to the best of my ability, but the interpretation of the suggestions is mine alone. Please check them. I have also altered the tag at the top of Talk:Science. I think that closes off that experimental review. I have also made sure the categories are working correctly. As regards the transcluded list going on the other Science WikiProject pages, my suggestion is that this be done the first time we have an article nominated for review in the area of a particular WikiProject and then add it to that WikiProject. Puting a list that only includes Computational Chemist on the Geology WikiProject page is not going to impress the folks there. --Bduke 01:26, 8 April 2006 (UTC)

[edit] Links on Science WikiProjects

To advertise new articles for review, I have started to add the following to the various Science WikiProjects:-

Start of inclusion.

[edit] Scientific Peer Review - Recent requests for review

This is a list of articles that have been nominated for scientific peer review. i.e. by scientists with expertise in the topic. Please contribute by adding new requests at the top.

Reviews of articles that have been nominated for review and the review completed are archived here.

End of inclusion.

Prompted by the new entry for History of Earth I have added it to:-

as well as Wikipedia:WikiProject Chemistry which I added it to because of Computational chemistry. Please add it to other WikiProjects pages. Note that the addition transcludes the list of articles for review, including a heading. --Bduke 02:39, 13 April 2006 (UTC)

Click the edit link that is not in the template - very important, otherwise you edit the stuff that gets transcluded in the main page and the project pages. --Bduke 22:41, 13 April 2006 (UTC)

The problem is that everyone seems a little apathetic about the project. I say let's just do something. Yes, get it out to the different WikiProjects. The problem is that they won't contribute without others, and those others won't contribute without them. It's the chicken and the egg problem. Hopefully this will help.--HereToHelp 13:03, 13 April 2006 (UTC)

[edit] Project directory

Hello. The WikiProject Council has recently updated the Wikipedia:WikiProject Council/Directory. This new directory includes a variety of categories and subcategories which will, with luck, potentially draw new members to the projects who are interested in those specific subjects. Please review the directory and make any changes to the entries for your project that you see fit. There is also a directory of portals, at User:B2T2/Portal, listing all the existing portals. Feel free to add any of them to the portals or comments section of your entries in the directory. The three columns regarding assessment, peer review, and collaboration are included in the directory for both the use of the projects themselves and for that of others. Having such departments will allow a project to more quickly and easily identify its most important articles and its articles in greatest need of improvement. If you have not already done so, please consider whether your project would benefit from having departments which deal in these matters. It is my hope that all the changes to the directory can be finished by the first of next month. Please feel free to make any changes you see fit to the entries for your project before then. If you should have any questions regarding this matter, please do not hesitate to contact me. Thank you. B2T2 14:19, 26 October 2006 (UTC)

Retrieved from "http://en.wikipedia.org../../../s/c/i/Wikipedia_talk%7EScientific_peer_review_80b8.html"