Wikipedia talk:WikiProject Good articles/Reform

From Wikipedia, the free encyclopedia

Contents 1 Initial comment 2 Process 3 Review process proposals 3.1 No change 3.2 Reviewer mentoring 3.3 Reviewer approval 3.4 Standardised reviews 3.5 Multiple reviewers 3.5.1 Proposal 1 3.6 Oversight 3.7 Specialised reviewers 3.8 Time limit 3.8.1 Statistics on quality 4 Review process proposals II - the winnowing 4.1 Open review proposal 4.2 General discussion

[edit] Initial comment

This page is intended to facilitate discussion of changes to the GA process. The front page is a place to present the issues, and ultimately, proposed changes, but I recommend that most discussions (ideas, proposals, thoughts) should take place on the talk page first. Geometry guy 19:28, 28 May 2008 (UTC)

[edit] Process

Maybe it would be useful to start by agreeing on an order to take these subjects in, to help avoid dilution and confusion on this page. I think possibly the review process or the GA criteria would be logical starting points, but what do others think? EyeSerene^talk 08:59, 30 May 2008 (UTC)

I'd probably vote for analyzing the review process first, since I personally don't have any major issues with the GA criteria. We did just implement a new system for moving GA reviews to subpages of article talk, which ties in to the process, so it might be good to start off with seeing how this is working so far and such. Also, I'll be reviewing GA reviews over the last two weeks for the 'reviewer of the week/month' award this weekend, and I plan to take another look especially into how well this transition to review subpages is going. Dr. Cash (talk) 14:59, 30 May 2008 (UTC)

Yes, those are pretty much my thoughts too - the criteria themselves seem reasonable to me, and I think any significant changes in them would be counterproductive and possibly even disastrous (thinking of Nehrams's 230ish swept articles that could end up back in the sweep queue...) EyeSerene^talk 16:17, 30 May 2008 (UTC)

AGGHH! Please don't change the criteria now! Just kidding, whatever we have to do to improve the process is fine, although I'm sure the articles swept by me and everyone else are still going to qualify as good articles if we were to change the criteria. I think the review process would be best to go over first as well. --Nehrams2020 (talk) 18:56, 30 May 2008 (UTC)

process first please. Let GA then stand for scrutiny then maybe a few changes might be necessary, might be though! I have yet to be convinced, but giving doubters etc the opportunity to follow GA reviews will hopefully answer many of their questions. Edmund Patrick ( ^confer _work) 20:08, 30 May 2008 (UTC)

Agreed. The "green dot debate" indicated that some in the wider community had concerns that that GA criteria were not suitable, since "substandard" articles were passing. I think it's more likely that the criteria is, by and large, appropriate, just not always applied rigorously (ie process). So, that's a good place to start; sort this out and we might find everything else slots into place with miminal hassle.

So, then, we have criteria for GA articles. We have articles submitted for assessment according to the criteria. What process would allow these articles to be measured against the criteria, in a consistent and appropriate way? Do we feel the current process is pretty much what we want (with minor tweaking), or does it need a complete overhaul? What do we think is the best way to review? Let's throw some thoughts down. Gwinva (talk) 20:24, 30 May 2008 (UTC)

How can we ensure that the community will accept the new process? OhanaUnited^{Talk page} 20:28, 30 May 2008 (UTC)

(remove indent) we cannot, we must just put it out there and let it stand, it is such an improvement that I think it will. ½ the points raised were I thought bought about by the "unaccountability" of previous GA reviews as well as the standard of those reviews. In a sense I felt one could not disprove the point because of the nature of the existing process. With sub pages of "contained" reviews hopefully some of the point can be answered by saying here is the history, here is the discussions! Then the criteria might need to be looked at. i will say that I feel that the subpages are a great step forward, one of those things that lead one to say "why didn't wew do this ages ago". Edmund Patrick – confer 20:38, 30 May 2008 (UTC)

We are merely here to thrash out ideas and create a proposal, which we will present to the community at large. The general feeling was that "something" needed to be done, but no one knew quite what. (Or, rather, a lot of people had ideas, but no conclusions were reached.) Here are some questions to get the ball rolling. (They are not original, but concerns/ideas I recall being raised during various discussions.) Do we like the one reviewer idea, or should it require more? A second opinion? An open process? Should it have a set time (ie no quick passes or fails)? Can anyone review or do they need approval? Organised review teams? A panel to approve the status once review complete? GA criteria checklist, or detailed reviews? → What do you guys think? Gwinva (talk) 22:56, 30 May 2008 (UTC)

OK, we seem to have consensus that we should be looking at the review process itself first. I've had a go at listing below each suggestion that has been raised here and elsewhere, in no particular order. As Gwinva says, everyone has their own ideas, so the list is unfortunately rather long (and I've probably missed some). I suggest we give a few weeks for comment, so that all interested editors can have their say, then start to winnow down the ideas into a few concrete proposals. EyeSerene^talk 08:24, 3 June 2008 (UTC)

[edit] Review process proposals

This list is not intended to be exhaustive, but as a starting point for discussion. Please comment, add and/or edit as you see fit.

[edit] No change

Having introduced review sub-pages and gone some way to addressing transparency concerns, are further changes still necessary?

[edit] Reviewer mentoring

Do we need to provide formal training to reviewers?

Comment - I like this idea. What I've been doing with the handful of GA reviews I've been doing is getting Giggy to check them over once I've finished to make sure I haven't missed anything, although sometimes I'm nervous about being overly harsh or critical. It's worked quite well for me and I'd reccomend it to anyone. Gazimoff ^Write_Read 15:54, 7 June 2008 (UTC)

[edit] Reviewer approval

Should only 'approved' reviewers be permitted to carry out reviews?

• Comment This would be the ideal solution to ensure consistent quality, IMO, although naturally it has potential problems. However, perhaps combined with reviewer training or mentoring, it would be possible to maintain a list of 'qualified' reviewers that are willing to be contacted on their talk pages to carry out reviews (much in the same way as the WP:PRV volunteers list). This could be instead of, or in addition to, posting nominations at GAN. EyeSerene^talk 09:07, 3 June 2008 (UTC)

• Comment I'd be more comfortable doing something like requiring two or more reviewers to sign off on a GA approval. I have found that when I have done GA review, the bad ones are easy to fail, but I don't think I've ever passed something I've done review on without asking someone else's opinion first anyway. Montanabw^(talk) 21:47, 4 June 2008 (UTC)

• "Comment2' I am not sure I like the concept of "approved" reviewers, would hate to have the RfA criteria applied to GA reviewers, who seem to be in short supply anyway, but on the other hand I also do think that there should be some way a reviewer does need to prove that they actually read and understood the criteria, maybe having an "apprenticeship" period where they can review articles, but need that 2nd opinion to pass. So, maybe not so much approval as training? Montanabw^(talk)

• Not a bad idea to have "apprenticeships". After so many reviews (needn't be high: just enough to see they've understood the citeria and there aren't any glaring issues), they can become "full" or "approved" or whatever. Gwinva (talk) 22:14, 4 June 2008 (UTC)
• I think this entire debate boils down to the question "how can we have faith in our reviewers?". There seem to be two basic solutions emerging - we either supervise them more closely (with open reviews, oversight or some such), or provide some sort of training then trust them to do a decent job. My preferred option is the latter, because it retains the one-reviewer, collaborative, unbureaucratic system that I believe is our greatest strength, and just seems more 'wiki' to me. EyeSerene^talk 10:37, 5 June 2008 (UTC)

[edit] Standardised reviews

Would changing the way the reviews themselves are formatted help with quality and consistency?

• We want to encourage detailed, helpful reviews, and discourage tick-boxing. Set formats might encourage the latter. But if they could be developed to encourage the former, then it might be worth considering. Gwinva (talk) 22:16, 4 June 2008 (UTC)

[edit] Multiple reviewers

Should reviews be carried out by more than one reviewer?

• Comment Reviews are invited by more than one reviewer under the current system, but it rarely happens from what I've seen unless someone requests a second opinion. Personally I'm not convinced about making this mandatory - we'd need to be careful about slowing down the process and making the backlog even bigger. I also think we should maintain GA's major selling point: the collaborative nature of GA reviewing, which may be diluted by multiple reviewers. EyeSerene^talk 08:49, 3 June 2008 (UTC)
• I like the idea of combining this with the "Reviewer approval" option... those who haven't been around GA for as long could take a review and have it "certified" (this sounds really formal but we can work on those kinks in time) by one of the more experienced folk, so two reviewers look at most (if not all) articles. dihydrogen monoxide (H₂O) 09:53, 3 June 2008 (UTC)
• I think I like the idea of a reviewer approval, though who becomes one and how could be fraught with difficulties. I am never sure if anyone followed my request for a second opinion, if they did presumably they liked what they saw and left. Edmund Patrick – confer 17:46, 3 June 2008 (UTC)
• I checked with the GA promotion process in a few other top-20 languages and they seem to have the multiple reviewers system. I suggest we can follow suit. OhanaUnited^{Talk page} 15:46, 4 June 2008 (UTC)
• Some kind of certification or approval process may be worth introducing, but I would be very much against the FA voting style of review. That would be the final nail in GA's coffin I believe. --Malleus Fatuorum (talk) 16:03, 4 June 2008 (UTC)
  • But how would we handpick those individuals for the certification process? OhanaUnited^{Talk page} 17:15, 4 June 2008 (UTC)
    • How were the first administrators picked? What about any editor who's reached some arbitrary number of reviews, 50 for instance, to kickstart the process? --Malleus Fatuorum (talk) 17:31, 4 June 2008 (UTC)
      • I have the same reservations as Malleus about creating an FA-lite system. However, the problem with using appointed/elected GA promoters (per Raul & Sandy at FA), is if we don't combine it with some sort of review voting process, any GA promoters won't be so much interpreting consensus, as acting on their own judgement. I don't think we want to end up with the perception that GA rests solely on the personal opinions of a few individuals. We could perhaps have an approvals panel, the membership of which rotates regularly, but I'm still unconvinced that's the best way to go. EyeSerene^talk 17:35, 4 June 2008 (UTC)
        • I'm very much against the idea of appointed promoters. I'm not even sure it always works that well at FA, never mind GA. This all, I think, echoes something that Geometry guy and Ling nut were discussing elsewhere. What is the purpose of GA? If it's to get as much of the encyclopedia as possible up to some minimum standard as quickly as possible, then it needs to be as process-lite as is consistent with some minimum qa. If it has some other purpose, then perhaps a more formal process needs to be considered. --Malleus Fatuorum (talk) 17:45, 4 June 2008 (UTC)

What we seem to like about one-article-one-reviewer is that it's much more efficient than having multiple people read the same article, and as long as we educate each other and look over each other's shoulders, we can probably trust the review if we can trust the reviewer. I am fine with the way we do it now; I would also be fine with any division of labor that doesn't involve significant overlap of responsibilities. Some people are better with process, some with language, some with content. - Dan Dank55 (talk)(mistakes) 19:05, 4 June 2008 (UTC)

I'd prefer we not have appointed reviewers as well. We do want to prevent that a new reviewer unfamiliar with the process doesn't approve a review by another inexperienced reviewer. This could rarely occur, but could still allow some articles to pass when they shouldn't. It might be best to ensure that the second reviewer has at least a minimum amount of articles they've reviewed to be able to serve as the second reviewer. --Nehrams2020 (talk) 19:07, 4 June 2008 (UTC)

I strongly feel that there are multiple reviewers. I find it rediculous that an article makes a semi-important list when there's only been the nominator and the possibly POV reviewer supporting an article. It's likely good, but there needs to be multiple reviewers. Reywas92^Talk 19:36, 4 June 2008 (UTC)

If I can pull something out of left field here, a recent (cough) contentious RfA has been generating a lot of discussion. My take is that humans like to have some kind of reasonable tests to pass, after which they are examined by their community and pronounced ready to "belong". My take with discontent with the RfA process is that the bar is set way too high for the RfA process to perform that function, so either the bar needs to come down a bit, or else people need to have other processes that are RfA-like but apply to smaller and more tightly-knit communities. Conceivably, the community of GA reviewers might benefit from an RfA-like process, and if that's true, then as a bonus, we maintain the efficiency of the current review process (one reviewer per article) while adding oversight and reliability that many people would like for GAN to have, rather than just saying "it's okay if stuff goes wrong, GAR will fix it". - Dan Dank55 (talk)(mistakes) 20:48, 4 June 2008 (UTC)

I think I broadly agree with all that. I hadn't considered the option of having some kind of RfA-like process to determine who can pass articles, perhaps due to my distaste for that process. But, if done properly ... perhaps. --Malleus Fatuorum (talk) 21:37, 4 June 2008 (UTC)

• Support I strongly support the concept of having two or more reviewers, but no more than three. Note my comments above in the "approved reviewers" section. Montanabw^(talk) 21:49, 4 June 2008 (UTC)
• Strong Oppose. Well, if we've started voting ... :-) --Malleus Fatuorum (talk) 21:50, 4 June 2008 (UTC)

Comment I've striken the two support & oppose votes. There is no vote going on in this discussion, yet. Dr. Cash (talk) 04:01, 5 June 2008 (UTC)

• I wouldn't like to see a voting system for GAs. The strength of the GA system is the collaborative dialogue which can take place between the reviewer and editor(s). So we need to place the emphasis on one lead reviewer. However, there are occasions when other opinions are useful, such in the case of novice reviewers, or where specific expertise adds another dimension. We also need to draw on the strengths of those involved in the project: some are good on image policies and reliable sources, but not so good at picking up copyediting issues (for example), others know physics inside out but wouldn't know where to start with TV. We also want to avoid situations such as inexperienced editors passing sub-standard articles (or those with glaring problems, such as image use), editors passing friends' articles in a "you scratch my back" kind of pact, driveby reviews, inconsistent application of criteria, poor understanding of subject-specific requirements etc etc, (ie. the current weaknesses) while still allowing/encouraging the GA strengths of collaborative dialogue and neutral non-wikiproject community recognition. I think a number of the issues overlap (time limits, reviewer approval, and so forth) but can be combined within one approach. I will outline a proposal of this kind below. Gwinva (talk) 22:54, 4 June 2008 (UTC)

[edit] Proposal 1

Proposal: When a reviewer selects an article to review, they effectively "open" the review (tagging it on GAN page with an appropriate tag), and become lead reviewer, carrying out a review in much the same manner as now. But the review stays open for a minimum period (say, 5 days). During this period, others may also add comments (constructive comments, not support or oppose votes).

These people might include: novices, who wish to begin their reviewing by contributing to open reviews; specialists, such as a scientist who has a quick look at every open scientific article to spot any glaring problems; experts, such as a copyright expert, who regularly goes through open reviews double checking image use tags; time-poor GA reviewers, who haven't time to contribute a full review, but want to be involved by offering helpful comments; mentor, who has agreed to oversee a new reviewer's first few reviews; concerned reviewer, who has some concerns about the approach of the lead reviewer and wants to keep an eye on things; 2nd opinioner, from a list of "senior reviewers" who has responded to a lead reviewer's invitation to provide a 2nd opinion in a difficult case (senior reviewers targetted especially, rather than vague "2nd opinion requested" tag on GAN page); random passer by, who spotted the GAN template on the talk page. But there is no requirement that others must contribute before a review is completed.

Once the minimum period has passed (regardless of how many have been involved in the debate), and issues have been resolved in some manner, then the "lead reviewer" closes the review, passing or failing the article accordingly. The lead reviewer should know the article and discussion well enough to make an appropriate judgement.

Explanation: This actually requires little change to the process. Many reviews, especially those carried out by experienced reviewers, will carry on much as they have been. But it allows for greater hand-holding of newbies, quality control, an elimination of "driveby" quick passes, an encouragement of small contributions, and a forum for using reviewers' strengths and expertise. Gwinva (talk) 22:54, 4 June 2008 (UTC)

• Comments:

• That seems like quite a sensible and workable proposal to me. --Malleus Fatuorum (talk) 23:01, 4 June 2008 (UTC)
• I have some reservations. Assigning the title of "lead reviewer" to whoever that opens the review could lead to argument and drama. It promotes a few individual to "hog" over and start a large amount of reviewers (e.g. more than five) at the same time just for the title of lead reviewer. It also leads to struggle for power. What if the lead reviewer disagrees with senior reviewer and both of them are not willing to compromise? We cannot let the lead reviewer to have a trump card and ignore everyone else. And although this process does have some quality control, it does not stop a determined user from promoting a poor-quality article to GA (especially when you say "there is no requirement that others must contribute before a review is completed"). I only feel that placing such a roadblock hinders the overall promotion process rate but do not see a large improvement in QC. OhanaUnited^{Talk page} 03:55, 5 June 2008 (UTC)

• "although this process does have some quality control, it does not stop a determined user from promoting a poor-quality article to GA" - nor does the current system. Nor does FAC, or any other process on Wikipedia. What are you proposing? giggy _(:O) 05:11, 5 June 2008 (UTC)

• I don't like the mention of "time poor GA reviewers". We shouldn't be promoting such "quick looks" at articles by reviewers that don't have time for a full review. This will only lead to sloppiness. You either have time to review the article or you don't. Dr. Cash (talk) 04:05, 5 June 2008 (UTC)

• The "lead reviewer"—he who does the main review, the bulk of it (I'd hope), wouldn't be the time poor one. That could just be if, say, you've got five minutes, and you notice a few typos that weren't brought up in the GA review. You add them on to the review because you don't have time to fix them yourself. giggy _(:O) 05:11, 5 June 2008 (UTC)

• Like Malleus, I like this idea. giggy _(:O) 05:11, 5 June 2008 (UTC)
• Our strengths under the current system are collaboration (especially in the 'training' of article editors), the personal touch, minimal bureaucracy, and (relatively fast) throughput. I know I keep banging on about it, but I think these are so important that, if our chosen solution minimises any of these, we'll do terrible harm to the project. Hence this proposal seems to be potentially the best so far, as it plays to our strengths - only the speed of throughput might be affected by doing away with quick-passes and fails, but I don't necessarily see this as a bad thing. Perhaps we could integrate reviewer training/mentoring into it somehow - maybe by asking new reviewers to sign up and assigning them a mentor, who would drop in on their first few reviews and give guidance where needed. I have a suspicion that in practice this system might still end up as one reviewer per article, but as long as they are trusted reviewers that's not a bad thing either ;) EyeSerene^talk 16:37, 5 June 2008 (UTC)

• I'm very much in agreement with what you say EyeSerene; we need to play to our strengths more than we need to pander to our critics. --Malleus Fatuorum (talk) 00:29, 7 June 2008 (UTC)

• Like Malleus, I like this idea. Very happy with the proposal. Edmund Patrick – confer 16:51, 5 June 2008 (UTC)
• This proposal actually makes a bit of sense. Though I would state it as keeping all articles 'open for review' for a minimum of three days. The person that initiates the reviewer makes his/her comments on the article, and anyone else that wishes to offer comments on it is free to do so. If no one else offers comments after three days, though, and the reviewer is satisfied that the article meets the criteria, then he/she is free to pass the article. If there's still a lot of work to do, keep it as-is (effectively "on hold", but the review is still "open"), or fail it. So effectively, we'd kill "quick fails", though if an article falls seriously short of the criteria, the nominator would have three days to address the issues and then it would just fail after three days.

Also, I'm not sure if we want to establish a heirarchy in reviewers, but we might want to strongly encourage newer reviewers to not initiate reviews on their own, instead focusing on making comments on open reviews, until they become more familiar with the criteria. Dr. Cash (talk) 17:22, 5 June 2008 (UTC)

This looks like a good idea, and like Dr. Cash, believe that a shorter period of three days would be better. If this was implemented, would we be able to create a subpage that automatically lists all of the articles currently on hold, so that reviewers could be better informed about what articles are open to a second look? I'm sure it could be updated by a bot. --Nehrams2020 (talk) 23:47, 5 June 2008 (UTC)

• I like it. - Dan Dank55 (talk)(mistakes) 23:54, 5 June 2008 (UTC)
  • P.S. This would dovetail nicely with some kind of vetting process. If someone fails even a lightweight vetting process, they could still help out, and also demonstrate that they're moving in the right direction, by helping out in reviews that don't "belong" to them. - Dan Dank55 (talk)(mistakes) 00:19, 7 June 2008 (UTC)
• I like the idea of supporting people who are new to the GA process and encouraging them to get a more seasoned reviewer to double-check their work. Gazimoff ^Write_Read 15:59, 7 June 2008 (UTC)
• I like this proposal. I don't see an issue with calling the initiator of the review the "lead reviewer". That doesn't come off as a hierarchy to me. Anyone can be a lead reviewer. It's just a matter of who chooses to initiate reviews and who chooses to help out with open reviews. The only "power" given to the lead is the responsibility of closing. Lara❤Love 19:06, 7 June 2008 (UTC)

• Great idea! The only problem I think is that the time is too long. 3 days ought to cut it as a minimum. If something either really, really good or really, really bad, IAR can lengthen or shorten the time as needed. bibliomaniac15 22:00, 8 June 2008 (UTC)
  • I don't have a preference how many days it is, as long as it's long enough so that everyone who wants to play some role in other people's reviews won't miss out because the window is too short. I would think a review that lasts only one weekend wouldn't be enough time to give everyone a chance to look at it, so 3 days seems a little short. - Dan Dank55 (talk)(mistakes) 22:46, 8 June 2008 (UTC)

I have made some modifications to the GAN review guidelines and put them on this temporary page. It essentially outlines a revised review process keeping all GA reviews open for a minimum time period of three days, or allowing a review to be passed/failed earlier if at least three reviewers agree on pass/fail. It also deprecates the on hold & second opinion statuses, letting reviews stay open for a longer period, if necessary (I put five to seven days here, but I don't think we want that to be a hard and fast rule).

If we implement this new procedure, the {{GAReview}} template should be modified to say something like, "Open Review – This review has been initiated by Dr. Cash (talk) 16:25, 9 June 2008 (UTC). Additional comments are welcome by other reviewers for three days." I would also recommend changing the icon used by {{GAReview}} to the same icon that is currently used by the "on hold" template, since it is more visible. Perhaps G-guy could figure out a way to link directly to the open review page from the link using some of his template magic? Also, if we implement this, all current on holds & 2nd opinions would be changed to the {{GAReview}} template.

Let me know what you think. Dr. Cash (talk) 16:25, 9 June 2008 (UTC)

I would have to disagree with the proposal, especially the requirement for three reviewers before an article can be quick failed. It gets too close to making GA like FA, and will leave GA clogged with articles that should be straight failed for massive issues. -- Collectonian (talk · contribs) 16:30, 9 June 2008 (UTC)

3 to quick-fail? That's not quick enough. 1 is enough to quick-fail, but I would say that the decision of quick-fail can be overturned in case of mistake or abuse. OhanaUnited^{Talk page} 02:49, 10 June 2008 (UTC)

The proposal looks fine except for having three reviewers in allowing a quick-fail. A single editor can determine this, and would only occur if the article was in such bad shape that it was no where near passing the criteria. I would think leaving it to one person is best, and as OhanaUnited stated above, we could allow for overturn with GAR if necessary. Quick-fails are not too common, but we don't want to continue to bog down our resources with requiring three (or even two) editors focusing on determining if an article merits a quick-fail. --Nehrams2020 (talk) 03:17, 10 June 2008 (UTC)

3 to quick fail is too many. I also think the 3 needed to pass is a bit much... I guess it depends who's reviewing, but in most cases 2 should be fine (what say we IAR? ;-)). giggy _(:O) 09:17, 10 June 2008 (UTC)

When the proposal is finalized, can someone type it out so that we can review the terms? I say that there is a lot of discussion and sometimes people (including me) just want to get straight to the point without the need to read an entire section. OhanaUnited^{Talk page} 14:14, 10 June 2008 (UTC)

Just to clarify, I think the above is referring to three days, not three reviewers, for article assessment. EyeSerene^talk 22:16, 10 June 2008 (UTC)

• I like the proposal. It subsumes the quick pass/fail process, which I have always found suspect, and simplifies the pipeline — there is now only one review process. The concept of keeping an article 'on hold' is subsumed by the notion that a review is open for a period of time. That other editors may participate in open reviews subsumes the '2nd opinion' concept. This proposal integrates the mentoring effort; editors new to the game may participate in open reviews as long as they care to and need not initiate a review of their own until they are comfortable with the ropes. Ditto for experienced reviewers who do not have time to administer a review, but who can offer commentary on other editors' open reviews. The concept of keeping a review open for a minimum period reduces the liklihood of erratic quality arising from quickly passed or failed articles. OhanaUnited raises the issue of editors 'struggling for power', which I take to mean as two or more editors disagreeing on whether an article is good or not. The default outcome of this lack of consensus, I suggest, would be 'fail' — not an unreasonable conclusion if editors fail to agree on the overall 'goodness' of an article. If particular editors battling over particular articles becomes epidemic, then I would suggest that editors who have significantly contributed to a review of a given article should henceforth be regarded as having contributed to the article itself, and, as such, should recluse themselves from future Good Article evaluations of that article. I'm not sure at what degree an editor's review becomes a 'significant contribution' to an article. Probably not spelling and grammar corrections, but probably if and editor has administered a review as lead, or has passed an opinion on whether the article passes or fails. Gosgood (talk) 13:49, 13 June 2008 (UTC)

[edit] Oversight

Do we need a mechanism to approve GA assessments before they become official?

• Comment: Maybe this would be the answer to the multiple reviewers/approved reviewers issue. Usually the poor articles are easy to fail, but another set of eyes before approval may be good. Montanabw^(talk) 21:50, 4 June 2008 (UTC)
• Comment - I think this would drastically improve quality. Unfortunately, we face a huge problem. Critics of GA cry that it's of inconsistent quality. However, any changes made to complicate the process and/or bring it closer to FA in anyway will be met with cries that it's redundant of FA. If it is agreed that such oversight is the best way to go, there are various options. Is it a panel of appointed GA veterans, or is it anyone with x number of reviews, who can approve reviews? And do they do a minor review of the entire article and accompanying review themselves, or simply scan over everything to ensure a complete review was carried out and complied with? Lara❤Love 19:11, 7 June 2008 (UTC)

[edit] Specialised reviewers

Would it help to have subject-expert reviewers that specialise in certain types of article?

• 'Comment: Maybe, though the view of a total outsider is good too. I think if there were multiple reviewers in some form, having one reviewer with a bit of expertise paired with an outsider might be the best of both worlds. Don't know how feasible that would be, but couldn't hurt to bat the concept around a bit. Montanabw^(talk) 21:52, 4 June 2008 (UTC)
• I've mixed feelings. GA is a community project, outside the wikiprojects (and the wikiproject assessements), which is a strength. However, specialists are more likely to pick up content problems, questionable sources, inaccuracies, and also be more familiar with subject specific MOS issues, or guidelines. A possible approach would be to have subject specific Ga criteria, so those unfamiliar with an area can get a quick idea of the specific requirements. Gwinva (talk) 22:10, 4 June 2008 (UTC)

Many such guidelines already exist; WP:FILMS and WP:UKCITIES spring to mind for instance. --Malleus Fatuorum (talk) 22:20, 4 June 2008 (UTC)

At present the articles are reviewed by volunteer wikipedians without any condition. I think in some cases the issues are completely technical and just an expert or somebody who is knowledgeable can judge correctly about them. While only one wikipedian review and judge about the article, it's better to use a knowledgeable wikipedian to do the task. Thus I suggest separating the technical issues and establish a technical review team, that is a group of wikipedians who are experts or knowledgable in some fields. We can also ask each wikiproject to introduce some knowledgable wikipedians.--Seyyed(t-c) 05:00, 3 June 2008 (UTC)

For example, please compare the two reviews about Sources of Islamic law. The first one has done by somebody who says I am by no means an expert in any sort of law, let alone Islamic law, but I think that will give me a great "outsider" perspective[1]and the second one has done by somebody who says I'm a Muslim and review the lead from Islamic viewpoint[2]. As you can see these are completely different viewpoints and in some cases we need both of them.--Seyyed(t-c) 06:31, 4 June 2008 (UTC)

This has been suggested several times before, including by me. The concept is fantastic and would most likely work. The problem is finding the experts willing to help. I've been out of this project for about 8 months (a message on my talk page led me here tonight), so I'm not sure what happened with my last few proposals, but if a page is created where experts can list themselves under their field of expertise and tandem reviewing can occur, I think that would help solve a lot of the problems GA faces with quality. Lara❤Love 07:00, 4 June 2008 (UTC)

I think we can ask the wikipedians who are active in wikiprojects. I think in some wikiprojects such as Military history we can find several reviewer. In some others like Wikipedia:WikiProject Sociology and Wikipedia:WikiProject Biology it may be difficult. But it's not a major problem, because few articles have been nominated which relate to such inactive wikiprojects.--Seyyed(t-c) 10:14, 4 June 2008 (UTC)

Some projects have too few members, while others may not have any members specialized in reviewing articles professionally. OhanaUnited^{Talk page} 03:51, 5 June 2008 (UTC)

As I told above the projects which have few member are inactive and we don't have problem with them. I think most of the articles which needs technical review are nominated by wikiprojects which are active and have enough specialized members in those fields. --Seyyed(t-c) 09:46, 5 June 2008 (UTC)

Yes, I think we'd have to closely integrate with the projects for this to work as a normal part of the review process. However, I don't think it's enough of an issue at GA, where we're concerned with broadness rather than comprehensiveness, that we need to worry too much about it. When expert opinions are needed - and it doesn't happen that often - it's fairly straightforward to request them, on a case-by-case basis, on the relevant project talk page. EyeSerene^talk 10:02, 5 June 2008 (UTC)

I think it ultimately comes down to reviewers knowing their boundaries in terms of what they can review and what they can't. For instance, I stick entirely within the parameters of reviewing History GANs, and rarely (if ever) step outside of that area, simply because that is my primary area of expertise. We should definitely make some sort of note to new GA-Reviewers to "remain within your knowledge boundaries". It might help to curb the instances shown above. Cheers! Cam (Chat) 21:19, 8 June 2008 (UTC)

I think that's about right. Video games, for instance. I don't like video games, I don't play video games, I know very little about video games, and I wouldn't consider reviewing a video game. --Malleus Fatuorum (talk) 21:30, 8 June 2008 (UTC)

• No harm in making this optional, but I don't think we should have a requirement of specialist reviewers. There are some really versatile people (Blnguyen is a good example) who shouldn't be held back by WikiProject membership. (And there are people like me who probably should, and so are by choice. :)) giggy _(:O) 10:06, 5 June 2008 (UTC)

• Hey, don't do yourself down giggy, I've got an opinion on just about everything as well. ;-) More seriously, I'll suggest that subject area experts are possibly the last people that you'd want to be involved in a GA review. --Malleus Fatuorum (talk) 00:40, 7 June 2008 (UTC)

• The last people involved? I'm thinking we disagree somewhat there - I'm curious as to why you think so? If it's because they might miss basic stuff, perhaps have a subject area and a... well... me (;-)) pair up to do the review? giggy _(:O) 02:44, 7 June 2008 (UTC)

• No, not because they might miss basic stuff, almost the opposite in fact. Because if you put two experts in a room, you'll get three opinions. --Malleus Fatuorum (talk) 17:00, 7 June 2008 (UTC)

• Am I correct in that the proposal is that GA reviewers do their review and, for the technical articles, they ask for input from someone with knowledge in the topic? In many cases, this would be unnecessary, as in cases where members of the Wikiproject are the ones that have done the work to bring the article up to standards. However, when that is not the case, I think it's important to bring in an expert before the article is passed. Lara❤Love 19:24, 7 June 2008 (UTC)

• That's how I see it, though I don't believe asking for expert opinion is always necessary. If a reviewer can't understand a technical article, they probably shouldn't be reviewing it anyway, and unless the expert is familiar with GA criteria we could find ourselves battling against unrealistic expectations of the criteria. The alternative might be to insist that all technical articles have already gone through a project A-class review prior to coming to GA, which I think has some merits. I'll wait to see what happens with proposals elsewhere before arguing that too hard though ;) EyeSerene^talk 19:43, 7 June 2008 (UTC)

• I disagree with the idea that if someone doesn't understand the technical aspect, they shouldn't review it. The entire point, for me, in tandem reviewing, is for two people with different expertise (technical aspect of the topic vs understanding of the GA criteria and the MOS) to review based on their expertise. One focusing on the technical aspect, the other on the MOS side. Having someone who doesn't understand the topic of the article is key, in my opinion. The point of our articles, in many cases, is to inform the uninformed. Having someone who doesn't know the topic ensures that the article is readable. Someone who already has an understanding of the topic is more likely to miss jargon and confusing prose, because they already know what they're reading about. Lara❤Love 19:50, 7 June 2008 (UTC)

• There's truth in that, certainly, but it does help to have at least, say, some mathematical savvy when reviewing mathematical articles. I don't subscribe to the view that we're writing articles to teach the uninformed; that's Wikiversity's job. Many technical articles will only be meaningful to readers with an in-depth subject knowledge, and I have no problem with that. I do believe there should be enough non-technical content (probably in the lead or introductory sections) that a layperson can understand why the subject is notable, but I'll leave it to the textbooks to do more than that. Hence my suggestion for post A-class reviews, when content will already have been verified by experts. EyeSerene^talk 20:02, 7 June 2008 (UTC)

• I agree a lot with LaraLove here - we're not trying to teach people, but we do want articles to be accessible to readers unfamiliar with the topics. Although wikilinks will help to a certain extent with defining words, I personally feel that being able to read and understand an article in isolation is a good thing. Because of that, having someone unfamiliar with the topic review an article can only be a good thing. I also think that a review is a two-way street, where the article nominator can discuss feedback with the reviewer in order to develop the article. Because of this, I'd expect the nom to feel comfortable with challenging the reviewer on areas they feel apropriate. Gazimoff ^Write_Read 20:50, 7 June 2008 (UTC)

• (outdent) I think there's a difference between accessibility and understanding. The article's prose should be of a high-enough standard that the article is readable, and explain why the subject is important and how it is used, but I think to expect a non-specialist to navigate to, for example, Hilbert space (which is currently a GA) and come away understanding the topic is to misunderstand what we are for. Per WP:NOTTEXTBOOK, we shouldn't really be in the business of instructing readers, just informing them. As you say, links can be used to follow up definitions and the like, but especially with technical articles I don't think it would be realistic to start from first principles every time. EyeSerene^talk 21:08, 7 June 2008 (UTC)
  • I think it probably should be... at least, the first principles should be made prominent somewhere (in conjunction with this). I don't think it's good practice to be giving the tick of approval to something a non-expert can barely make meaning out of. I suppose you're not saying this either... in any case, I think we agree that a non subject expert should be able to make some sense of the topic after reading the article. The specialist reviewer—the subject expert—should be able to point out factual inaccuracies in the article that I, for one, would never pick up on. giggy _(:O) 01:39, 8 June 2008 (UTC)
    • Precisely. That, in my opinion, makes tandem reviewing perfect. It covers the accuracy, broadness and technical details. The GA reviewer focuses on the style aspect. Ensuring the lead is well-constructed, the punctuation is correct throughout, images are compliant, etc. Together the two reviewers and the article custodian(s) can work to ensure the prose flows well while maintaining accuracy. As far as requiring such articles be at A-class before receiving GA, I just don't think that's going to happen. FA is the step that generally follows A-class, which is considered higher on the rating scale than GA. Lara❤Love 05:22, 8 June 2008 (UTC)
      • Surely the spirit of GA is that the article's authors/nominators are the experts? And that those subject-matter experts work in tandem with a GA reviewer? I'm afraid this proposal just seems like another move in the direction of FA-lite to me, another nail in the coffin. --Malleus Fatuorum (talk) 21:36, 8 June 2008 (UTC)

Is "FA-lite" a real threat? I guess I mean, don't we have a firm consensus that we don't more than one or two people to be reading the same GAN article, unless people just want to train or vet or learn from the main reviewer(s)? That seems like the opposite of the FAC philosophy to me. And on the other point, a collaboration between the reviewer and the editors as "content experts" is nice when it happens, but sometimes they're not experts. - Dan Dank55 (talk)(mistakes) 22:39, 8 June 2008 (UTC)

I believe that FA-lite is a real threat, although I'm beginning also to believe that I'm perhaps the only one who does. As for the article's authors not being being "experts", well, as I said above, if it's not a topic that you feel you know enough about to review, then don't review it. --Malleus Fatuorum (talk) 22:50, 8 June 2008 (UTC)

So do I, as it happens. I think we can just ask around the projects on those (infrequent IMO) occasions when an outside subject expert is needed and no-one in the GA community has the requisite knowledge to fill in, or as Malleus says, leave the review to someone who knows a bit more about it. We've run into difficulties in the past with GA criteria being applied inappropriately to technical articles (as Lara will remember better than I, having had to deal with the fallout!), sometimes expert advice will be needed, but I agree that, in general terms, we can still assess against the criteria for layout, MoS, prose, NPOV, image copyright etc, and the authors should supply the content. EyeSerene^talk 12:12, 9 June 2008 (UTC)

FA-lite is a real threat. And I warned against this somewhere on this page, I believe. If you make improvements to the GA project, you're going to hear "It's becoming redundant of FA" from opposers of the project. If you don't make improvements, you're going to hear "It's a poor quality project worth nothing." This is why I left the project last year. It was lose/lose. We made several major changes to the project that, I believe, significantly improved the quality. But the squawking from the opposition was steadfast. However, I've had time to reflect, and I've come to the following conclusion:

We have two options. 1/ We leave the project as is, knowing there is need for improvement and hearing it regularly from those who oppose the GA project, or 2/ We make improvements and receive the customary "They're moving more inline with FA; GA is redundant of FA, delete it!" arguments, the project survives because, let's face it, there's support on this project to keep GA, and the end result is an improved project. Short version of my conclusion: Who cares? Everyone here is a volunteer in a self-appointed position. We put ourselves in GA, so let's do what we do and not worry about what those looking in from a distance have to say unless they offer valid concerns for which we can build upon. Lara❤Love 12:16, 9 June 2008 (UTC)

I see that a lot of GAN reviewers provide lists of their reviews on their userpages. Surely a history of reviews that were later confirmed at GAR or FAC (or at least, confirmed with respect to the GAN criteria) suggests that the reviewer is competent and trustworthy? Maybe we should be more explicit on our userpages that we're not just boasting, that pointing out that our judgment has been confirmed is an essential part of making GAN more efficient that FAC, by allowing and encouraging public scrutiny of the reviewers. I don't have anything bad to say about FAC reviewers, but it does seem to me that having one reviewer instead of 10, in addition to being 10 times more efficient, is also braver and shows a willingness to take personal responsibility for one's work. - Dan Dank55 (talk)(mistakes) 15:16, 9 June 2008 (UTC)

[edit] Time limit

Would changing the period allowed for reviews help? (for example, by insisting all reviews are posted for a set period)

• I don't think there should be a hard and fast rule regarding a time limit for reviews (on hold status). Granted, we don't want reviews to take forever, and I think most will take about 5-7 days or so. But if it takes a little longer to get it done, and as long as the editors are responding to comments and actively editing it, it's fine to take longer. It's probably best to stick with the status quo here and let reviewers find the best times for reviews/on holds themselves. Dr. Cash (talk) 21:40, 3 June 2008 (UTC)
  • Agreed, hold periods under the current system should be flexible where constructive work is ongoing (within reason!). Do you think it would help to leave other reviews (ie outright passes, quickfails) up for a set period too, so no assessment is complete until it's been open for comment for, say, one week? EyeSerene^talk 08:11, 4 June 2008 (UTC)
    • A minimum time limit would be a good idea; it would go some way to eliminating the "drive-by" reviews that occasionally occur. This would basically rule out "quick passes" and "quick fails", and (hopefully) encourage more detailed/specific reviews. A one line "yes it passes" is as unhelpful as "sorry, it fails"; most people submitting for GA want feedback as well as green dots. A time limit would also allow some level of quality control: it doesn't need to be a full "multiple review", so much as an opportunity for others (if inclined) to cast an eye over things. Better to spot problems (with reviews) at this stage, rather than waiting for GAR. Gwinva (talk) 21:57, 4 June 2008 (UTC)
• I disagree with a time limit, and always have. I've passed holds in less than an hour, and I've had holds go for over a month. If the article is being worked on, or if there is a promise of work on the article (author has exams/is on holidays/whatever) soon, there should be nothing arbitrary holding the review back. giggy _(:O) 05:13, 5 June 2008 (UTC)
  • That said, some newer reviewers may benefit from a set time (at least 24 hours?) to have someone else take a quick look over their work, in conjunction with one of the other proposals gaining some weight here. giggy _(:O) 05:13, 5 June 2008 (UTC)

I think we should rely on the reviewer's judgment. Reviewer can wait more, if he/she finds the editors are active to improve it and the problems are minor. On the other hand he/she can fail it soon when editors are inactive or problems are serious. --Seyyed(t-c) 11:21, 5 June 2008 (UTC)

Let the reviewer decide, but still recommend a week if possible. This will hopefully prevent the editors of taking their time to getting around to fixing the article, allowing for more time on focusing on new articles. We don't want to rush the editors, but don't want them to take an extreme amount of time that isn't necessary. I've passed articles in a matter of a few days or a few weeks, based on the editors continually working on improving it. --Nehrams2020 (talk) 23:50, 5 June 2008 (UTC)

• I wouldn't remove the option for quick-fails. There are some articles that just don't come close to meeting the standards. Much to often the backlog at GAN is over a month. If we lose the option to quick-fail articles that are obviously below the standards for GA, the backlog is only going to grow. I haven't looked recently, but the quick-fail criteria may still be in need of tweaking. As far as a minimum time-limit on reviews and holds, I wouldn't make it more than 3 days for reviews. I'm not a fan of drive-by reviews, but when I was in the mood to review, I could to a thorough review in a matter of a couple hours. However, whether a pass or fail, a couple more days for others to possibly look could never hurt. For holds, I think seven days minimum is good. I, like giggy, would always extend as long as progress was being made. Lara❤Love 19:29, 7 June 2008 (UTC)
• I think the merit in getting rid of quick-fails is that since we're trying to drive up quality, it might help to ensure that one-line fail notices become a thing of the past. Although we're not intending to review an article, it doesn't hurt to leave a decent explanation of why, and what can be done to fix it (as we would after a normal 'fail' review). Nice to see you back, BTW! EyeSerene^talk 19:48, 7 June 2008 (UTC)

[edit] Statistics on quality

It would be helpful to get an idea of what the quality of our reviews currently stand at. I'm not sure if any statistics are being gathered from sweeps, but it could be valuable information. I also think it important to attempt to get some information from SandyGeorgia regarding what the common issues are with recently promoted GAs that she finds at FAC. If she has any information about what the most common issues are, or even the most common reviewers to pass deficient articles, it would give us a good starting point for project improvement in regards to reviews. Lara❤Love 19:31, 7 June 2008 (UTC)

[edit] Review process proposals II - the winnowing

There has been little movement here for a few days now, so can I take it we're all discussioned out and it's time to move on? I've tried to distil down the comments that have been made so far, and summarise the state of play. Again, if I've left anything out, or over-simplified/misrepresented anything, please make corrections as you see fit ;) EyeSerene^talk 18:37, 12 June 2008 (UTC)

Unsupported

• There was little support for maintaining the status quo, although it was pointed out that whatever we do, we will not change the minds of all GA's opponents, so we may be better off simply ignoring calls for change. There was also little enthusiasm for putting in place some kind of official approval mechanism for signing off on GA reviews - whilst perhaps a good idea in theory, there seem to be too many potential problems to make this a practical solution. The idea of addressing review quality by insisting all reviews follow a standard format was also largely unsupported. Voting on GA assessments was generally disapproved of, and felt to be too similar to the FA assessment system.

No consensus

• The suggestion to team up with subject experts when conducting reviews reached no real consensus in how it should operate; although it was widely agreed that expert advice concerning article content is very useful and sometimes indispensable, how this should be obtained remains undecided. It was noted that GA reviewers should expect the article authors, as subject specialists, to provide content advice where needed, and reviewers can in any case assess most of the GA criteria without any need to fully understand the article content. A system for 'approving' reviewers was thought by some to be useful, but there was no agreement on how this could be achieved without over-bureaucratising the project.

Supported

• The remaining proposals, after some debate, crystallised into broad agreement on a slightly different review mechanism to that used at present. Incorporating the proposals for more than one reviewer, an adjustment to the review time-limit, and the guidance of new reviewers, general support was expressed for the idea of 'open' reviews. These could be initiated and closed by a lead reviewer but open to anyone interested enough to comment, and could remain open for a minimum set time period. It was felt that this would address the concerns with review transparency and drive-by reviewing, and provide a means for new reviewers to 'wet their feet' by joining in with in-progress reviews before taking the lead role in subsequent ones. It would also not preclude single-reviewer reviews, collaboration, and a personal approach, thus retaining the strengths of the current GA process. However, some of the details remain under discussion.

Since only one proposed solution emerged, I've set this out below, point by point, and indicated where there may be a need for further discussion. I don't think there's any need to turn this into a vote (it's pretty much a one-horse race), but if anyone wants to add alternative proposals, please do!

[edit] Open review proposal

• Reviews are open for anyone to leave comments.

Consensus reached

• Reviews are opened and closed by an established 'lead' reviewer (the reviewer who initiates the review).

Consensus reached

• All reviews remain open for a minimum period (no more quick passes/fails).

No consensus as to how long, though between 3-5 days seems to be the preference

• 3 days is fine; after all this is a minimum, and the lead reviewer may choose to keep it open, in much the same manner as an "on hold" now. 3 days is probably enough for all concerned to have a look; but I am not averse to 5 days. I don't think we need to be concerned about "clogging up" the GAN page with open reviews: when people wait a month or so, another couple of days won't make much difference. I see this as replacing the current "on hold" status.Gwinva (talk) 21:52, 12 June 2008 (UTC)

• There is no requirement for reviews to have additional reviewers.

Weak consensus, although some support for tandem reviewing, perhaps with a subject expert

• I see this as case-by-case. At the moment we get good reviews from experienced reviewers, and we don't need to complicate those by adding extra requirements. But some mechanism for requesting extra opinions (from experts or experienced reviewers) might need to be established. This might be flagging the review on the GAN page (like "2nd review requested") or by the lead reviewer issuing an invitation to a list of mentors/subject experts. Gwinva (talk) 21:52, 12 June 2008 (UTC)

• New reviewers are encouraged to participate in current reviews before taking on the lead role.

Consensus reached; may need to discuss how this should be monitored (if at all)

• I don't think we need to formalise anything beyond the current mentor programme (I'm assuming the mentors will wish to keep an eye on their protege's first few reviews). Merely clarify the "review instructions" to suggest new reviewers start by contributing to open reviews, and a pointer to the mentor programme. Some reviewers may feel confident starting out on their own, and we shouldn't discourage that, or devalue real-world experience. Gwinva (talk) 21:52, 12 June 2008 (UTC)
  • If we say "Wait a while until you take the lead role", it puts the new reviewer in the position of having to guess how long we want them to wait. I'm happy with what Gwinva is saying, but I think it goes hand-in-hand with some kind of light-weight approval process after, say, the new reviewer has done 5 of their own reviews. - Dan Dank55 (talk)(mistakes) 23:20, 12 June 2008 (UTC)

• Review status can be monitored.

More discussion needed; possibilities include listing open reviews somewhere, categorising them, changing the way they're listed on the GAN page etc...

• There have been some criticisms recently about the complicated "paperwork" of the GA process. An easy categorising or bot-assisted process would be good. One radical idea might be to dispense with the GAN page altogether (or turning it into a instruction page). The GAN template can categorise from status. "nominate" → Category:Ga noms (by subject); "open" → Category:open reviews; "failed" → Cat:Failed GAs; "passed" → Cat:GAs. Reviewers go to Cat:GA noms rather than the GAN page when they feel in the mood for reviewing. Or Cat:open if they like looking at open ones. Status changes take place on the article talk page, rather than all this listing and commenting at GAN. Gwinva (talk) 21:52, 12 June 2008 (UTC)

[edit] General discussion

Thanks, EyeSerene for taking this forward. I've bolded the above points, to make them stand out, and added a few comments. Seeing it laid out here all looks very positive; I feel we are not far off a workable proposal. Gwinva (talk) 21:52, 12 June 2008 (UTC)