Wikipedia:WikiProject Reforming Wikipedia/Expert peer review

Outline

The vast majority of articles are written by amateurs. In many topic areas, well-written, accurate, and comprehensive articles can still be produced without expert attention. However, in the trickier areas, such as in mathematics, science, and philosophy; that have a large body of knowledge, truth, or opinion, amateurs may face difficulty in bringing articles up to quality; oftentimes, these articles contain fundamental errors or omissions.

There are a wide range of solutions to this problem, of which the most radical is to close editing altogether in these tricky topic areas and allow only experts to edit. This is likely to be rejected by the WMF and the community, goes against our open-editing philosophy, and will likely lead articles to become out-of-date without many editors (see also Scholarpedia, where article quality is high, but breadth is low). We will also have trouble attracting experts, as it is difficult to provide an incentive to edit. A better solution would be to implement a system of expert peer review, inviting experts to review articles and afterwards recognizing them for their work, which is the main motivator towards experts to review.

Expert peer review can not only help improve Wikipedia's accuracy and usefulness to readers, it can also help heal the notorious reputation of Wikipedia as unreliable, as well as improve academic perceptions. Readers are more likely to trust Wikipedia if we can demonstrate serious efforts to improve our articles, and academics' view of Wikipedia will improve as we invite them to have a say on what many readers rely on daily.

Who is an expert?

For the purposes of this proposal, an expert is any person who has an advanced degree in their field, such as a PhD; and:

has published, at the very least, several papers in their field in an independent periodical or other work that have been cited at least once.^[a] There is no quantitative standard for this requirement; editors should assess each person on a case-by-case basis.
has won a notable award in their field (such as the Abel Prize or Fields Medal for mathematics). Membership in honor societies or other organizations (such as Phi Beta Kappa)^[b] or honorary degrees are not considered awards.

How can one prove that they are an expert?

For a reviewer to prove that they meet this criteria, they should link to their Wikipedia account (either made after registration or an active editor) on one of their personal webpages or on their resumé or curriculum vitae (which should be accessible on the Internet). No other proof will be accepted. This is required to assuage any concerns relating to the Essjay controversy, as the controversy could have been easily avoided if we had asked Essjay or any other editor claiming false credentials to prove their identity. Experts should also be willing to disclose their name and institution (if any).

What is the scope of this system?

The following process is intended to have as light an effect on community consensus and control over content as possible, as well as a supplement to the good and featured article status. For this first proposed trial, peer review can be requested on existing good or featured vital articles; or as part of good or featured article nominations of vital articles, in order to assist Wikipedian reviewers in making their decision on whether to promote or fail the article for GA/FA, but not as a formal requirement. These groups of articles are eligible:

The scope will be extended if the trial is successful.

How will peer review under this system work?

Several volunteers are needed to invite experts to review article requests and to coordinate with the expert reviewers; any editor may act as a volunteer providing that they follow the process below.^[d] The peer review process, from request to completion, proceeds as follows:

An editor requests that an eligible article be reviewed on a dedicated request page. good articles and featured articles, as well as articles that have been nominated for those statuses, are eligible for review.^[e] The editor must be a primary editor of the article or otherwise discussed the prospect of review with the article's primary editors to avoid any misunderstanding or controversy. The primary editors of the article should edit the article based on the expert's suggestions; if they are inactive, the requesting editor or the assigned volunteer to the review task should pick up the responsibility.
A volunteer picks up the request, and beings searching for suitable experts (note that this can be semi-automated). Volunteers should try to find the best expert for the task, sending invites via email to possible candidates.^[f] Volunteers must remain formal and professional; experts will most certainly respond unamicably to casual requests.
If an expert accepts the offer to review, the volunteer should direct them to create an account with an appropriate username. The volunteer should setup the review, informing editors working on the article, creating a review page, and teaching the expert about basic wiki markup.
- If the expert does not wish to create an account, but instead wishes to send a document containing the review (i.e., PDF, Word, TeX, etc.), then the volunteer should not abandon the review. The volunteer should act as an intermediary between the expert and the editors, relaying both parties' comments.

What will be shown to readers on peer reviewed articles?

A previous version of this article on 31 December 2099 was peer reviewed by experts.

Summary · List of reviewers · More information

For the trial, a template will be placed only on the talk page. But if this proposal and the renewal proposal succeeds, then we will place a template on the main article similar to the one on the right.

Notes

^ This requirement exists to prevent obvious fringe theorists, and authors of blatant quackery. There are no concrete cutoffs for this requirement.
^ Except if the organization or membership is considered highly prestigious, such as fellowship in the Royal Society for a specific field.
^ Mathematics degrees are without prejudice to whether they were given after completion of a pure or applied mathematics program, although it is preferable to seek a mathematician with a specific degree in different cases (e.g., it would be preferable to seek a pure mathematician to review Banach–Tarski paradox).
^ Unless the expert is already an experienced Wikipedia editor; in which case, they may implement their changes directly. It may even be advantageous to not go through this process if they wish to review an article themselves, as they can handle any disagreements over their suggestions easily, while a new or inexperienced expert editor will need guidance.
^ Quality articles are reviewed regardless of the time since the previous GA or FA review; in fact, newly-promoted articles are preferred because a review is likely to yield improvements that non-expert reviewers have missed.
^ Additionally, this means that they should avoid inviting disreputable "experts", such as fringe theorists and academic fraudsters.

[1] This requirement exists to prevent obvious fringe theorists, and authors of blatant quackery. There are no concrete cutoffs for this requirement.

[2] Except if the organization or membership is considered highly prestigious, such as fellowship in the Royal Society for a specific field.

[3] Mathematics degrees are without prejudice to whether they were given after completion of a pure or applied mathematics program, although it is preferable to seek a mathematician with a specific degree in different cases (e.g., it would be preferable to seek a pure mathematician to review Banach–Tarski paradox).

[4] Unless the expert is already an experienced Wikipedia editor; in which case, they may implement their changes directly. It may even be advantageous to not go through this process if they wish to review an article themselves, as they can handle any disagreements over their suggestions easily, while a new or inexperienced expert editor will need guidance.

[5] Quality articles are reviewed regardless of the time since the previous GA or FA review; in fact, newly-promoted articles are preferred because a review is likely to yield improvements that non-expert reviewers have missed.

[6] Additionally, this means that they should avoid inviting disreputable "experts", such as fringe theorists and academic fraudsters.

[a]

[b]

[c]

[d]

[e]

[f]