Wikipedia:External peer review/Nature December 2005

Nature (December 2005)

A complete list of errors with their current status can be found at Wikipedia:External peer review/Nature December 2005/Errors

• Source: Nature
• Date: 14 December 2005
• Title: Internet encyclopedias go head to head
• URLs:
  • Nature blog: Comparing Wikipedia and Britannica
  • "Internet encyclopaedias go head to head"
  • Nature reviewer reports (pdf) - link dead, contents available at Wikipedia:External_peer_review/Nature_December_2005/Errors
  • List of articles reviewed (doc)
  • Nature editorial
• Related news articles:
  • BBC News
  • Business Week
  • Aljazeera.net
  • USA Today
  • Scientific American blog
• other links at Wikinews
• Britannica rebuttal
  • "Fatally flawed: refuting the recent study on encyclopedic accuracy by the journal Nature" (PDF). Encyclopædia Britannica, Inc. 2006. Retrieved 2006-03-23.
  • Nature mag cooked Wikipedia study - Britannica hits back at junk science - By Andrew Orlowski in San Francisco
  • James Bone Britannica 'still rules' over web rival in The Times March 25 2006

Nature compared Wikipedia and Britannica science articles and sent them to experts in the field. The number of "factual errors, critical omissions and misleading statements" were recorded.

From their blog:

We're trying to see if we can publish the full list of errors found by our reviewers, or least send them to you (and to Britannica if they want). We'll post an update here as soon as we have a firm answer.

I also received a private email from them in response to a request for more information. I hope they don't mind me posting it below:

In light of the amount of interest, we have decided to make the reviews public as far as possible, although obviously we'll have to edit them to remove the names of the reviewers, any libellous statements etc. The reviewers didn't all respond in the same format, and some of them highlighted points that we didn't consider to be significant errors, so we're also writing up an accompanying document to explain which errors we counted, and how we arrived at all the numbers. We're also asking the reviewers if they mind being identified, so we'll name those who give permission. That's all quite a bit of work, especially with Jim being away, but I hope we can send this to you by the end of next week, as well as putting it up (free) on our own website. Thanks for your patience! (15 December 2005)

Update: The reviewer reports are now available on the Nature web site, in Microsoft Word format. See above URL.

This review was covered at Wikipedia:Wikipedia Signpost/2005-12-19/Nature study

Findings

Of the 42 articles reviewed, 38 were found to have at least one error – Britannica had 40 articles with at least one error. (NOTE: Nature took some of the excerpts for the study from the version of Britannica for children and youths rather than use the official version for adults. Britannica claims the study is invalid as a result because none of the articles taken from Wikipedia were "for children or youths.") — Preceding unsigned comment added by 64.134.40.134 (talk) 00:02, 25 November 2012 (UTC)

The following articles had the highest number of errors:

• 19 errors
  • Dmitry Mendeleev
• 11 errors
  • Cambrian explosion
• 9 errors
  • Paul Dirac
• 7 errors
  • Acheulean industry
  • Neural network
  • Prion
  • Thyroid
• 5 errors
  • Ethanol

The following articles had no errors highlighted:

• Subrahmanyan Chandrasekhar
• Lipid
• Punctuated equilibrium
• Quark

Nature's special report also noted the following:

• "several Nature reviewers" found the Wikipedia article they reviewed to be "poorly structured and confusing" — a criticism that the report notes is common among information scientists;
• unnamed information scientists also "point to other problems with article quality, such as undue prominence given to controversial scientific theories" (see Wikipedia:Guidelines for controversial articles and Wikipedia:Neutral point of view for guidelines related to this problem);
• In Wikipedia's defense, Michael Twidale, an information scientist at the University of Illinois at Urbana-Champaign, told Nature that Wikipedia's strongest suit is the speed at which it can be updated, a factor not considered by the journal's reviewers.

Update: the detailed reviewer reports are now available (see above).

Errors per word comparison

Please post below a table of errors/word statistics, based upon the Nature article and the word counts in the corresponding articles, so that we can see a more controlled comparison of error rates. —Steven G. Johnson 02:53, 15 December 2005 (UTC)

If it seems to make sense to calculate and compare the ratio errors/words, it probably makes less sense to compare the ratio omissions/words (1 omission in a 5000 words article might be considered to be more serious than 1 omission in a 1000 words article on the same topic, whereas 1 error in 5000 words is probably better than 1 error in a 1000 words); and it might make even less sense to mix both categories in an attempt to gauge any bias in favour or against Wikipedia (or Britannica) in the Nature experiment. I would therefore be very careful in drawing any conclusions based on the numbers in the table below. --83.180.100.151 20:48, 8 January 2006 (UTC)

Britannica displays the word count for each article, doesn't it? At least that part shouldn't take long. ‣ᓛᖁᑐ 03:29, 15 December 2005 (UTC)

You'd think that not all words are created equal. — Ambush Commander^(Talk) 03:43, 15 December 2005 (UTC)

No doubt, but this seems the quickest reasonable statistic to gather. —Steven G. Johnson

Is this really more controlled? The average WP article has lots of fluff, with no space-pressure to remove same. Errors/omissions-per-article seem a reasonable metric to me. +sj + 07:37, 15 December 2005 (UTC)

Your fluff is another person's interesting tidbits. =) Honestly, it's hard to make a quantitative assessment of information content, but if there is a large difference in article length then it is a hint that apples are not being compared to apples. (Nature claims that the article lengths were comparable, but I'm finding this hard to reconcile in some cases, e.g. West Nile virus (see below) where the WP article has apparently been almost 5 times longer than EB's for a year now.) The size differences seem large enough that I'm inclined to think that the 30% difference between EB and WP in the Nature study is washed out by systematic problems, although that of course depends on the type and severity of the errors. —Steven G. Johnson 08:35, 15 December 2005 (UTC)

Note that, for a fair comparison, we shouldn't include tables of contents, external links, "see also", or references — most Britannica articles do not include these, and the Nature review did not consider referencing quality. Nature refers to "factual errors, omissions or misleading statements", so some of the "errors" listed below may be errors of omission in incomplete articles, rather than factual errors.

Article name	Britannica			Wikipedia
	Word count	Errors	Errors/word	Word count	Errors	Errors/word
Acheulean industry	500	1	0.002	417	7	0.016787
Agent Orange	252	2	0.00793	1270	2	0.0015748
Aldol reaction	130	4	0.030769	660	3	0.0045455
Archimedes' principle	350	2	0.0057143	607	2	0.0032949
Australopithecus africanus	235	1	0.0042553	496	1	0.0020161
Bethe, Hans	658	1	0.0015198	1823	2	0.0010971
Cambrian explosion	519	10	0.019268	702 (13 Dec.)	11	0.0157
Cavity magnetron	394	2	0.0050761	1121	2	0.0017841
Chandrasekhar, Subrahmanyan	365	4	0.010959	417	0	0
CJD	591	2	0.0033841	1373	5	0.0036417
Cloud	641	3	0.0046802	1689	5	0.0029603
Colloid	561	3	0.0053476	896	6	0.0066964
Dirac, Paul	837	10	0.011947	1044	9	0.0086207
Dolly	1334	1	0.00074963	807	4	0.0049566
Epitaxy	178	5	0.028090	235	2	0.0085106
Ethanol *	315	3	0.0095238	2631	5	0.0019004
Field effect transistor	588	3	0.0051020	933	3	0.00322
Haber process	241	1	0.0041494	531	2	0.0037665
Kinetic isotope effect	210	1	0.0047619	569	2	0.0035149
Kin selection	923	3	0.0032503	404	3	0.0074257
Lipid	349	3	0.0085960	676	0	0
Lomborg, Bjorn	518	1	0.0019305	1501	1	0.00066622
Lymphocyte	479	1	0.0020877	351	2	0.0056980
Mayr, Ernst	357	0	0	753	3	0.0039841
Meliaceae	152	1	0.0065789	281	3	0.010676
Mendeleev, Dmitry	1306	8	0.0061256	1134	19	0.016755
Mutation	728	8	0.010989	1557	6	0.0038536
Neural network	557	2	0.0035907	1233	7	0.0056772
Nobel prize	409	4	0.0097800	2052	5	0.0024366
Pheromone	313	3	0.0095847	461	2	0.0043384
Prion	473	3	0.0063425	1583	7	0.0044220
Punctuated equilibrium	943	1	0.0010604	1265	0	0
Pythagoras' theorem *	688	1	0.0014535	1899	1	0.00052659
Quark	1112	5	0.0044964	2060	0	0
Royal Greenwich Observatory	235	3	0.012766	532	5	0.0093985
Royal Society	416	6	0.014423	869	2	0.0023015
Synchrotron	770	2	0.0025974	1590	2	0.0012579
Thyroid	583	4	0.0068611	1459	7	0.0047978
Vesalius, Andreas	930	2	0.0021505	1174	4	0.0034072
West Nile Virus	245	1	0.0040816	1320	5	0.0037879
Wolfram, Stephen	475	2	0.0042105	559	2	0.0035778
Woodward, Robert Burns	873	0	0	2320	3	0.0012931
Total	22733	123		45254	162
Mean	541.26	2.9286	0.0054106	1077.5	3.8571	0.0035798

* - Articles marked as good articles

• Word counts were computed by pasting text from the Firefox browser into the Unix wc program. This gives a slight over-estimate because it counts anything surrounded by whitespace as a "word". Britannica counts were taken from Britannica Online. Wikipedia articles were from 14 December 2005 except where otherwise noted.
• Tables of contents, external links, see also, and reference sections were excluded from word counts.

As a note of caution, since Nature said the lengths of articles they compared were roughly equal, the versions they compared must be different from those whose length we compare here (since our versions are now longer than the EB articles.). This isn't surprising considering the lead time needed to select the articles, send them out for review, gather the reviews, and compile and publish the results. - Nunh-huh 06:43, 15 December 2005 (UTC)

I find that a bit dubious. By that theory, we should be able to date the review by going back to the last time the articles were comparable to the EB articles (which presumably haven't changed much recently). However, I just checked the Vesalius, Andreas article (which is one of the egregious examples where WP is 10 times the word count of EB), and you have to go back to 2002 to get significantly shorter than it is now, which seems unlikely. I find it more likely that the editors simply tossed out any obvious stubs or near-stubs. —Steven G. Johnson 06:55, 15 December 2005 (UTC)

So something is amiss. I somehow doubt Nature compared a 62 word EB article on Robert Burns Woodward to his Wikipedia 2300 word article. (And found no significant omissions in EB!) - Nunh-huh 06:59, 15 December 2005 (UTC)

Oh @#$#@, no it's my fault. I miscounted the EB Vesalius article — it's one of the (very few) EB articles which is spread over multiple pages, and I only counted one page. Similarly with the Woodward article. I'll go back and recheck any others where the imbalance seems to be large. (Update: counts should be corrected now.) —Steven G. Johnson 07:02, 15 December 2005 (UTC)

If there winds up being a significant difference, then the explanation is most likely error or a difference in versions. Nature specifically states "All entries were chosen to be approximately the same length in both encyclopaedias. In a small number of cases some material, such as reference lists, was removed to make the lengths of the entries more similar." [1]. - Nunh-huh 07:24, 15 December 2005 (UTC)

Note that the references, external links, and "see also" sections were excluded from all of my counts. I still find discrepancies I can't explain. Our West Nile virus article, even one year ago, was 1108 words or 4.5 times the word count of the EB article. I've searched around EB, and I can't find any huge alternative article on this virus that they could have used instead. —Steven G. Johnson 07:31, 15 December 2005 (UTC)

Is there an alternative EB website? Are we counting the words in an EB "junior" site? Just a possibility. - Nunh-huh 08:08, 15 December 2005 (UTC)

According to their website (to which my university gives me access), I am searching the full EB. (They also have "student", "concise", and "elementary" versions of EB, but I'm not using those.) I'm not accusing Nature of dishonesty, but I admit I'm mystified. I wouldn't be surprised if they thought a factor of two was "comparable" length, but a factor of almost 5 seems like a lot. —Steven G. Johnson 08:14, 15 December 2005 (UTC)

I've emailed the news editor and asked if they could provide the exact article date/time versions that they sent to those 50 experts. That should be useful for figuring out if the errors have already been fixed, and for comparing the versions they sent and the soon-to-be more accurate ones. It's possible that they didn't think to check their versions, but they probably did. I'll pass on the reply when it comes. --Mr. Billion 23:09, 16 December 2005 (UTC)

No reply. Ah, well. We'll see. --Mr. Billion 19:32, 18 December 2005 (UTC)

From the editors (see supplementary Nature report above):

Each of the reporters that worked on the survey chose 10 to 15 scientific terms that were roughly in their scientific beat – the sorts of things we ourselves would check in an encyclopaedia. We had not looked at any of these entries in either encyclopaedia when we selected them. Then we weeded out the terms that did not have any entry in Britannica (they all appeared in Wikipedia), and any for which the entries were vastly different in length. Sometimes the lengths were balanced by amalgamating two or three Britannica entries into one coherent piece – for example, 'ethanol' was done this way. We felt this represented 'everything Britannica had to say on the subject' – at least, everything we could find by a quick search of Britannica online, exactly the way a user would approach

So, the criterion was not "vastly different" in length, which would allow e.g. a factor of two difference, and maybe even a factor of 5. —Steven G. Johnson 17:35, 22 December 2005 (UTC)

Article status

None of the articles reviewed have featured article status, and none have undergone our internal peer review process. Two articles, Ethanol and Pythagorean theorem, have good article status with the latter being a former featured article (see here, though there is virtually no discussion there). violet/riga (t) 19:48, 15 December 2005 (UTC)

Note that Aldol reaction is now tended by a few PhD Chemistry candidates and is FA.--SupperBird 22:19, 3 October 2007 (UTC)

Tagging every Nature-reviewed articles w/errors an overreaction?

Putting a "This article has been identified as possibly containing errors" tag on the front page of each and every articles identified by Nature seemed to some to be a bit of an overreaction. Others were quite comfortable with the practice.

---

In the long run, it seems that the project survived the tagging of all articles in which Nature's experts found errors. I would suggest that the tagging was even constructive and usefu. I note that we tag tens of thousands of articles as part of the Category:Wikipedia backlog by rank amateurs and teen-agers with éclat. Perhaps our amateurs are simply uncomfortable being reviewed by experts — just as Jimmy Wales finds working with domain experts to be "intimidating".

References:

1. The Trends Underlying Enterprise 2.0 March 24, 2006. Quote: "Nupedia’s 7-step peer review process was heavily biased toward Ph.D holders and other alleged "true experts in their fields," and was evidently elaborate and daunting.
2. After failing to make an article contribution to the failed Nupedia proeject, Wales explained years later that even he was intimidated at the thought of submitting an article (in economics: his area of training and experience) to it because was intimidated that experts (rather than amateurs) would review it.Interview: Knowledge to the people 31 Jan 2007.--76.204.176.148 22:15, 3 October 2007 (UTC)

Correction progress

While some progress was made in the days immediately after the report came out, the effort stalled as the Christmas 2005 holiday approached. When one examines the progress made before Christmas, one can see that much of the work was "low hanging fruit" in the form of easy corrections to superficial criticisms or where specific corrections were offered by the expert reviewers.

To be clear: it took several weeks before even 50% of the listed corrections were made. Unfortunately, some Wikipedia commentators like David Weinberger were under the false impression that "almost all" of the corrections were made within the first 24 hours of the effort. See Web of Ideas: The Authority of Wikipedia Around minute 70 of audio track. March 17, 2006. This further propagates the misconception prompted by the fact that Wikipedia has millions of accounts registered. More realistically, Wikipedia has only a few thousand active editors (or ten thousand if you use a generous definition of "active editor"), with varying levels of knowledge and maturity and many of whom contribute little to mainspace articles outside of reverting obvious vandalism. Refer to Wikipedia:List of Wikipedians by featured article nominations which lists appromixate 1000 users for a realistic list of contributors who have ever contributed substantial amounts of original quality content (with perhaps a factor of 5X for substantial sub unlisted collaborators). There are about 1000 administrators, and less than 100 other users with more elevated access. Also, in the new master plan, there will be only about 2000 "trusted" users, but all users will be rated with new "reliability" software to be implemented. Of course, one's personal relationship with Jimmy Wales or one's popularity within the community will still be able to override this number.

If one examines the status at the start of 2006, one sees the pattern: easy fixes or articles with a small number of errors were addressed at the start of the effort. Many of the easy fixes were biographical or narrative in nature and of a "coffee table book" level of understanding typical of an English major. Note that only the Aldol reaction article was a Featured article at the time of the review and that this article is fortunate to have three PhD chemistry candidates as caretakers (having such expertise involved on an extended basis is a rare situation at Wikipedia). The remaining articles required a conceptual understanding of chemistry, math, physics and engineering at around the college undergraduate level. The criticisms of articles such as Archimedes Principle, Dmitri Mendeleev, Colloid, Epitaxy, Field effect transistor, Kinetic isotope effect, Prion, Thyroid were not just syntactical corrections: they required an understanding at the semantic level on these scientifically non-controversial and well-understood topics for which there are, in an absolute and objective sense, right and wrong answers that are not intuitively obvious but require the ability to master technical concepts and facts.

For the academically honest, these later corrections were not subject to public opinion or Wikipedia's style of consensus building akin to voting since most people (and probably most Wikipedians, including the Board of Trustees and past and present members of the Arbitration committee, other WikiMedia foundation advisers and personnel and the employees of Wikia) do not and never will understand these concepts correctly and with depth. One user in particular, Pinktulip, did demonstrate such intellectual mastery and made a steady single-handed effort during January 2006 to ensure that another month did not pass before all the corrections were accomplished. Pinktulip also then informed Jimmy Wales of the completion of the overall effort on 25 January 2006.

Note that Wikipedia:Expert retention is an ongoing problem at Wikipedia.--Simongar 20:14, 26 September 2007 (UTC)

Britannica's Reply

Britannica has replied to Nature's article and claim the nature article was itself innacurate in a number of ways. It would probably be good for Wikipedians to check the sources Britannica gives and note needed changes in our articles.

Nature's Response

Nature has now issued comments on Britannica's rebuttal, and stands by its original article. [2] (archive).

Jim Giles made a presentation at Wikimania on 2006-08-04.