User:TidyCat/Achieving validation on Wikipedia

Comments on this essay are warmly welcomed on the Talk page. I have added a full copy of it there so users may comment on it point-by-point if they so wish. --TidyCat

Hi. I've been editing Wikipedia for a few months now (and reading it for far longer), and it is a wonderful project. The Wiki model is astonishingly effective, and I am one of many people who believe in its world-changing potential.

But like many others, I've also encountered Wikipedia's key shortcoming: in its present form, it is of questionable use as a reference source. It is free, it is large, and it is an excellent source of information on many subjects. But the information it contains can be so variable in quality that Wikipedia does not, and cannot have, authority. At any given moment, any article on Wikipedia can be incomprehensible, inaccurate, one-sided, vandalized, or even wholly fictional, possibly with malicious intent. This devastates Wikipedia's central goal: to be an encyclopedia.

Initial research

I started studying the problem in depth; for several weeks I've been poring over the encyclopedia, contributing, learning about its working methods, and reviewing its history. I learned that the concept of article validation goes back almost to Wikipedia's inception -- and that many proposals to implement it have been submitted, but none have yet taken hold.

In the midst of my research, the Seigenthaler incident occurred, and since then, criticism has been coming at Wikipedia fast and hard. These incidents have dramatically underscored the need for article validation on Wikipedia -- no longer as a "should," but as a "must."

There have been several validation schemes suggested on Wikipedia, and many of the ideas have been quite good. I think their failure to be adopted to date has been a combination of a lack of urgency -- which is now upon us -- and the absence of certain key elements in their specification.

Since the Seigenthaler incident, there has been a renewed interest in validation, and there are several proposals being actively discussed. I have attempted to draw the best elements of each of these proposals together into a full and practical specification which still aims to keep the Wiki as open as possible. I hope the advocates of these various proposals can see that I am neither ignoring nor taking credit for their ideas.

It is my goal to rally the community to adopt a workable and full solution in the very near future. If there is any one project that needs prioritizing on Wikipedia, this is it. Let us make this long-sought feature a reality this time.

Notable proposals

Most of the proposals I've seen fall into a few major categories:

• Expert review
• Voting/rating of individual articles/versions
• Community review (à la Peer review or Featured article candidates)
• Voting/rating of individual editors (including "trust metrics")
• Stable/locked versions

All of these approaches face the challenge of needing to satisfy two seemingly-opposed goals: to keep Wikipedia's openness intact -- anyone may edit Wikipedia -- while enacting a process that will weed out misinformation.

Any approach to article validation must thus satisfy the following criteria:

• Avoid the "gatekeeper" effect -- there should be no privileged users whose ability to shape the content of Wikipedia far exceeds others';
• Scale to accommodate potentially hundreds of editors' inputs without descending into chaos or inaction;
• Be implementable upon tens or hundreds of thousands of articles within a reasonable time frame;
• Be resistant to abuses by individuals or groups trying to subvert the system; and
• Produce articles which are reliably factual, consistent in style, and neutral in tone.

Expert review

Expert review -- the appointment of credentialed academics to shape Wikipedia's content -- fails on several of these points. There are many active Wikipedians who are experts in a number of fields, but it seems unlikely that they, working alone, could keep up with the huge volume of edits and new articles on the site without becoming a bottleneck to improvements. Hiring reviewers would likely be prohibitively expensive.

A greater danger is the possibility that such experts would edit articles according to what they know or believe, and discard whatever falls outside of their personal knowledge.

Because the standard embraced by Wikipedia is to only include statements based on independently verifiable sources, expert review is in fact not even necessary. The only expertise reviewers would need is the ability to understand the article they are reviewing -- and the sources from which it is drawn. Wikipedia's expert contributors have been invaluable, and will continue to be so, but the project has been built on far more than their efforts.

Voting on articles

Article voting or rating is a passive process, and it seems unlikely that it could ensure Wikipedia articles reliably met specific objective standards. Rating also is an inherently subjective process: while I am sure many Wikipedians will vote thoughtfully, I suspect most will simply vote along the lines of "I like this," "I hate this," "This seems correct," and so on.

It is my understanding that a system of this type is close to being implemented on Wikipedia. I do believe it will be of use in evaluating certain aspects of articles' quality, and it will probably help draw attention to a variety of articles which could use improvement. But it is at best an indirect mechanism for rooting out misinformation in the encyclopedia, and it is conceivable that it will be little better than the existing mechanism -- "You can edit this page right now!" I mean all due respect to its proponents -- but an article rating system should not be considered a true mechanism for validation.

Community review

A formal review process is probably the most direct way to evaluate and standardize articles according to objective criteria. Any article on Wikipedia could be brought up to encyclopedic quality by a team of motivated Wikipedians. But the question is how to implement it: Who reviews? Who edits? What standards should be applied? What procedures should be followed?

Featured Article Candidates is an excellent example of a process by which Wikipedians organize themselves into review teams and turn out excellent articles. However, the challenges that an official validation process would face are likely to be different from the Featured Article process, so certain changes would need to be made.

Voting on editors

If a review process were adopted on Wikipedia, those performing the reviews would have a strong shaping influence over article content. Wikipedia would need to be able to trust the reviewers to ensure that articles met all quality standards, as well as to ensure that articles incorporated community views.

A "trusted-user" system already exists on Wikipedia: the promotion of certain editors to adminship, based on their contributions to the project and to the community. It is not a perfect system, but it seems to have worked pretty well and produced a large staff of admins who are for the most part conscientious and able.

A similar procedure could be adopted to create a broader staff of "reviewers": individuals who could be trusted to carry out the task of evaluating each article according to objective, community-developed standards -- and who can serve as effective mediators of the review process, making sure that all viewpoints are considered, and that consensus is achieved.

Stable/locked versions

No matter which validation method is ultimately adopted, Wikipedia should provide a way to view the exact version of the article that passed validation. This is critical: there is no point to offering readers a validated version if we can't guarantee that the version they view is the version that was actually validated.

This could be accomplished by providing a link to a specific version in the article's history, or by placing validated versions on a separate page which is protected from edits. I favor the separate-page approach, as it will place validated versions in a consistent, easy-to-find location, and allow for greater versatility in the presentation of validated versions.

The Featured Article review process

A typical Featured Article candidate is nominated and sponsored by a single editor. Several self-appointed reviewers offer comments and suggestions, and the article's sponsor(s) do what they can to bring the article in line with the reviewers' suggestions. On occasion, reviewers will help rewrite parts of the article.

This system works well for a variety of reasons: the participants in Featured Article reviews are generally like-minded individuals; they avoid reviewing articles with a contentious history (including edit wars); there is usually only one sponsoring editor, avoiding differences over style or content; and the end result has no particular "official" status -- it is featured on the Main Page for one day, but the overall result is merely a good article having the same official status as every other version of every article on Wikipedia.

FA status is prestigious, and a Featured Article is generally well written and researched. But the FA process, as it currently works, doesn't satisfy all of Wikipedia's needs for article validation. For one thing, readers of a given featured article are not informed that it has passed the FA process (and thus has a higher likelihood of being a good source of information). FA status is noted on articles' Talk pages and on the master list of Featured Articles, but neither of these will come to the attention of most readers.

More importantly, after an article has become featured, the article is not stabilized in any way -- a link to the reviewed version in the article's history is not even usually provided. This means that even a Featured Article may be a vandalized article at any given moment that a reader encounters it.

Therefore, the FA process would need to be altered in a handful of ways in order to cover all the requirements that an "official" validation process would need to satisfy.

Proposed article validation process

An official validation process on Wikipedia would need to address the following issues which the Featured Article process currently sidesteps:

• It must be able to produce balanced and reasonably well-written articles even on contentious subjects;
• The process must be open to the input of a potentially unlimited number of editors; and
• The end result should be a stable, although not final, version.

The validation process must thus be part standardization and part negotiation. There will be many differing views on how to change each article to bring them in line with community standards, particularly as regards the requirement of neutrality.

The FA process offers clues as to how this might be accomplished. FA reviewers are generally not the authors of the article, nor do they perform whatever rewrites are needed. They state their opinions as to where each article falls short of established standards, and leave the actual rewriting to others.

This approach usually allows them to evaluate the article more objectively than any of its authors could. Furthermore, they are compelled to make whatever objections they might have clear, specific, and actionable -- "I don't like it" is not actionable, whereas "This section is unclear/unsupported by references" is.

In order to broaden this approach to accommodate multiple parties actively writing the article, only one simple change would need to be made. Instead of reviewers making their recommendations to a single editor (the article's nominator/sponsor), they should make their recommendations to all editors with an interest in the article. Any editor could submit to the reviewers their version of how they would rewrite a given section. Reviewers would then decide amongst themselves which version was the most readable, factual, or neutral (depending on which objections needed to be satisfied).

Editors, in turn, could offer their own suggestions as to what changes they felt the article needed. Reviewers would be compelled to incorporate these suggestions, as long as they fit within established criteria and met with few objections from participating editors.

A negotiation process of this type is already applied throughout Wikipedia as the basic mechanism for achieving consensus. By combining it with a "mediator class" (the reviewers), it ought be possible to achieve consensus (or as close to it as possible) on every article on Wikipedia.

Proposed validation criteria

Featured Article candidates are evaluated according to a set of standard criteria. These criteria are, at the present time:

• The article should be well written;
• comprehensive;
• factually accurate (with included references);
• neutral;
• stable (not subject to ongoing edit wars);
• compliant with community standards of style;
• have legally-usable images, where appropriate; and
• be of appropriate length, remaining focused on its main subject (referencing "daughter" articles where necessary).

Most of these criteria should also be applied to articles undergoing validation. However, where Featured Articles are intended to meet (or exceed) standards for "Wikipedia 1.0," validated articles should merely be required to meet the "0.5" standard (usable articles) mentioned on the Wikipedia 1.0 page.

I thus propose that validation criteria should be modified from the FA criteria in the following ways:

• Validated articles should be reasonably well written. As long as they are readable, reasonably structured, and cast in an encyclopedic "voice," that should be sufficient.
• Articles should not have to be comprehensive. As long as the article is not completely one-sided in its coverage of a subject, it should be acceptable. More roundly-written articles, however, might be higher-priority candidates for validation.
• Articles should not have to be stable. A lot of articles have been the focus of bitter edit wars, vandalism, and bias. Many of these articles need a validated (i.e., stable) version more badly than less controversial subjects.

In essence, validated articles should not strive for perfection. The primary purpose of validation should be to prevent the dissemination of misinformation. As articles improve through user contributions and successive reviews, they should be held to higher and higher standards -- each new validated version should be an improvement on the prior one. But the initial standard for validation should be a relatively relaxed one (except where factuality is concerned). The best mechanism for improving articles' overall completeness and quality is the Wiki process itself.

Nevertheless, there will probably be cases where articles will be too incomplete or otherwise flawed to be meaningfully validated. Reviewers in these cases could decline to review an article and instead suggest it be listed on the improvement drive or peer review pages. Featured Article reviewers commonly do this, directing articles that fall far short of the FA standard to Peer Review first.

In addition to the above relaxations of the FA criteria, I would suggest the following additions:

• References should not just be required, but explicitly checked by at least one reviewer. I am sure many FA reviewers do check the references provided in the articles they review; however, it is not presently a requirement that these sources be checked, and just as a statement can be false or erroneous, so can a citation. Explicit fact-checking is possibly the most important aspect of article validation.
  This requirement might be difficult to meet in cases where a reference cannot be checked online. If an alternative reference can be found which supports the statement(s), it should be cited instead, or in addition.
• Plagiarism should also be checked for explicitly. FA reviewers commonly do this, particularly with images; however, if feasible, simple checks for plagiarism should be performed on the text as well (at the very least, a Web search).

The Featured Article process, meanwhile, can remain on Wikipedia without modification. Where validation should strive to give us many articles meeting the 0.5 standard ("usable"), Featured Articles should continue to push articles to the 1.0 standard and beyond ("our best work").

Post-validation

After an article passes review and is formally validated, it should be placed on a "Validated" page, accessible via a tab placed beside the current article tab. The Validated and Current articles should have a prominent link to each other at the top of the article, with a note explaining the difference.

The Validated version should be locked from edits. This is critical: there is no point to offering a reader a validated version if we can't guarantee that the version they view is the version that was actually validated. Ongoing improvements to the article, however, should continue on the Current page in normal, unrestricted Wiki fashion.

It might be useful to give the Validated page its own "Talk" page, where users could list specific errors they believe exist in the validated article, possibly prompting a re-review. (Since it would have this specialized purpose, the page should perhaps be called "Errata," and its tab could be labeled "Report an error." The top of the Errata page could additionally suggest "If you like, you may edit the current draft of the article and fix it yourself," with a link to the Current article.)

At any time, any article on Wikipedia could be nominated by any editor for review (or re-review). However, there should probably be a moratorium of perhaps a month between reviews to keep ideological battles from being waged constantly over certain articles. (Editors could continue to tilt at each other on Current and Talk pages.)

In cases where a user could demonstrate a specific error in a recently-validated article, the review process could be reopened to correct that error only. If the review team can be shown to have been grossly negligent in enacting community standards or incorporating editors' inputs (hopefully this will be rare), the article could be opened for a full re-review. And in cases where substantially new material were added to the current article (particularly material covering current events), the review could be reopened to include the new material only. Outside the moratorium period, however, all articles should be fair game for re-review.

I'm not sure what mechanism should be used to determine if an article in moratorium merited re-review: some editors might constantly submit material of low relevance and insist that it be included. One possibility might be a short-duration straw poll of reviewers only (possibly requiring a supermajority). This could permit the inclusion of new material without keeping the article in constant dispute. Disputed material could still be added to the Current page, and if moratorium periods were kept short, a full re-review could start in the near future anyway.

A similar straw poll approach might be necessary during a full review to determine whether disputed material should be included in the main article, moved to a subarticle, or discarded altogether. I am not advocating voting over each minute aspect of each article, but there will probably be issues over which consensus through discussion cannot be achieved.

Default display version

Having both Validated and Current versions of articles available on Wikipedia raises an issue: which one should users be shown, by default?

There are sound reasons for each version to be the default shown to Wikipedia visitors. The Validated version will have a high likelihood of being of good quality -- at a minimum, factual and free of vandalism. However, the Current article may contain far more information than the most recent Validated version. Much more importantly, Wikipedia should do everything it can to attract new editors to the encyclopedia. If visitors encountered a "working draft," that might inspire more of them to contribute than if they saw a stable version.

However, I feel the potentially edit-suppressing effects of stable versions could be mitigated in several ways. For one thing, links to view and to edit the Current version should always be provided, with a prominent banner explaining the difference between the versions offered to unregistered users. Registered users should be given the links between versions but would not need the explanatory text. Registered users should also be able to set whether they want to see Current or Validated versions by default.

Another mitigating factor is that there are only two cases where a "default version" would need to be shown -- when a user types in an article name and presses the 'Go' button, or types its URL directly into the address bar. In cases where a user performs a search (either from within Wikipedia or via an external search site), both versions should have equal standing, and both versions could appear in the search results, depending on how closely they matched the search.

A third method of visiting articles -- clicking on wikilinks found in other articles -- would not need to favor a version either. If the user is presently viewing the Current version of an article, its wikilinks should take the user to the Current version of linked articles. If the user is viewing the Validated version, its wikilinks should load the Validated version of the linked article, if one is available.

The election of reviewers

The Requests for Adminship process works fairly well in identifying editors who can be counted on to carry out administrative and regulatory tasks according to established rules. A similar system could be created to elect reviewers -- individuals who could be counted on to carry out reviews competently and impartially.

The standards for reviewership could probably be far less restrictive than the standards for adminship. For one thing, reviewers will have a very limited, although important power: the ability to evaluate whether an article meets community standards, and make recommendations as to how it should be changed to meet those standards. For another thing, reviewers will never act alone, but instead as part a team of reviewers, who must achieve consensus in their evaluation of articles.

The standards for achieving adminship are not formalized, although most voters on admin candidates apply a high standard. The standards for reviewership would also not need to be formalized -- editors may apply their own standards for whom they think would make a good reviewer.

Admins should be allowed to be reviewers, but should not be granted reviewership automatically. The duties of admins and reviewers are very different, and a person who is suited to one role may not be suited to the other. In keeping with the existing policy that discourages admins from exercising admin powers upon articles they actively edit, they should also be discouraged from performing admin actions on articles they actively review.

It is inevitable that some reviewers will prove to be bad mediators, possibly ignoring others' views or applying personal (not community) standards of style. There will thus need to be some regulatory mechanism for demoting reviewers who abuse their powers. I feel that a remedy which would not require admin or arbitrator involvement would be preferable.

One possible approach could be to have reviewers elected to finite terms of service, of perhaps 3 to 6 months. Reviewers could apply for re-election so as to serve continuously, if they so wished. I feel this would be a good mechanism by which editors' suitability as reviewers could be periodically re-evaluated, without burdening administrators or arbitrators with the problematic duty of deciding "who's in" and "who's out." I hope that only extreme cases of abuse would necessitate the attention of administrators or arbitrators.

The proposed review process, step-by-step

A lot of the working details of how a review should proceed will probably emerge in response to practical concerns. To get the ball rolling, however, I would like to propose an initial model for the validation process.

Initiating a review

Any editor may nominate any article for validation review. There are several ways the nomination procedure could be handled. One simple way of handling it could be to add a template to the article. The template could do any or all of the following things:

• Announce to readers of the article that it has been nominated for review;
• Provide a link to a "Validation review candidates" page where they may second the nomination by signing under a heading containing the article's name;
• Add the article to a category, "Validation review candidates"

Reviewers could then sign on to review the article on the "review candidates" page. Alternatively, they could leave comments there -- for instance, to suggest that an article needs to cite its sources, or that it is too short to be a high-priority candidate for review.

Once enough reviewers agree to review the article (at least two, but perhaps more), the reviewers should open a "review" page specific to the article. The simplest way to implement this would probably be to add a "Review:" namespace to Wikipedia, which could be accessed via a tab at the top of each article. This would also make it easy to let interested editors know when a review of a particular article has begun: they could simply keep its Review page on their watchlist.

Once a review has begun, the article should be removed from the "review candidates" page and added to a "current reviews" page.

Opening steps of a review

To open the review, reviewers should copy the current article (or a suitable recent version) to the Review page. The Review page should serve as a repository for agreed-upon article changes and should be editable by reviewers only. The discussion of what changes should be made should take place on a different page -- most likely another namespace, "Review Talk."

The "Review Talk" page is where reviewers should discuss their objections to any part of the article, or other changes they wish to see. All editors may respond on this page with their own opinions, or by offering proposed implementations of the changes.

The "Current" article page, meanwhile, should be locked from edits for the duration of the review. The reason for this is to keep the current article and validated article from getting out of sync while the review is being performed. If the end result of the review -- the validated article -- were copied to the Validated page but not the Current page, that would produce a content fork between the two pages, which is highly undesirable. The validated version should be used as the starting point for all edits which follow the most recent review.

However, if the validated article does get copied to the Current page, then any changes editors make to the Current page during the review will get discarded. Rather than allow them to waste their time, a template should be added to the Current article directing them to the Review Talk page, where they may contribute to the active review.

Because the opening of a review would involve the locking of an article's Current page, an admin should be the official initiator of the review proceedings. This would also ensure that the criteria for review are met: namely, a minimum number of committed reviewers, and the article not in "moratorium" from a recent review. To commence a review, an admin should add the "review in progress" template to the Current page and protect it. The review can then proceed.

Sequence of review

To avoid performing work that might be discarded, the article should probably be reviewed according to a specific sequence. The presence of plagiarism should be checked for first; then citations checked and factuality disputes resolved; then major structural issues tackled, such as the regrouping of passages in new sections, or the removal or addition of material as necessary to keep the article balanced or focused on its core topic.

General issues may then be addressed -- essentially, any objection a reviewer wishes to raise. Non-reviewers may also raise objections -- if they are reasonable objections, reviewers should attempt to address them; otherwise, they may be disregarded. It may be useful to address articles on a section-by-section basis until every issue is finally resolved.

Following such a sequence should be a practical guideline, not a requirement, as every article will potentially have unique issues that need resolving in different ways.

Closing the review

It might be necessary to halt or abandon a review -- if the reviewers fail to actually conduct the review, for instance, or if events of any other sort impede its completion. If that happens, the review should be cancelled, and an admin should reopen the Current page to edits.

However, provided the review successfully addresses all of the major issues raised, the revision assembled on the Review page should be given one last fact-checking and proofreading, and an admin should be notified of the successful conclusion of the review. The admin should briefly review the proceedings to confirm they were conducted properly, then copy the Review version to the Validated page as well as to the Current page. On the Validated page, all templates should be subst'ed, and all images used should be locked from edits (if necessary, a copy specific to the article could be used). Finally, the article listing should be removed from the "current reviews" page. The Current page can then be reopened to edits by all users.

Technical changes

In order to implement the proposals I have listed above, the following technical changes would need to be made to the Wikipedia site and/or the MediaWiki software:

• The addition of Validated, Review, Review Talk, and Errata namespaces;
• The addition of tabs and messages to the basic interface which direct users between each of these namespaces;
• The possible renaming of the Article tab to "Current" or something similar;
• New code to load either Current or Validated pages via wikilinks, depending on whether the user is currently viewing a Current or Validated page;
• The addition of a user option to view Current or Validated pages by default; and
• The addition of a "Reviewer" user class which can edit the Review page.

There might be other necessary technical changes which I have not considered.

Closing thoughts

I do feel that a true system of article validation can be implemented on Wikipedia without disturbing the openness of the Wiki. Not all of the solutions I've presented here may be the best ones, and not everyone may agree with them. However, I do feel that validation's time has come, and that we can fit together all the necessary pieces in the near future, allowing Wikipedia to become not only the world's most comprehensive source of information, but its best.

I welcome your thoughts on the Talk page.

--TidyCat 05:00, 8 January 2006 (UTC)