User:Doc James/Rebut
Authors
[edit]James Heilman Ian Furst
Is Wikipedia’s Medical Content Really 90% Wrong?
[edit]The journal article by Hasty et al. published on May 1st 2014 [1] claims that most Wikipedia articles on important medical topics contain a large number of errors. The claim is based on a study of ten articles as they appeared on April 25th 2012. In nine of these the authors noted differences from a peer-reviewed source. The popular press, fueled by interviews with Hasty, has taken this to mean that as much as 90% of Wikipedia's medical content is wrong.[2] [3] [4] However, we found that Hasty et al. used an unvalidated test of content quality and applied it to Wikipedia alone. Additionally there were significant errors in study design and data analysis. The authors' conclusions are not supported by the results.
Ten osteopathic students or residents were each assigned two Wikipedia articles to evaluate. Each Wikipedia article was assessed by two students to determine how many assertions of fact they contained, and whether those statements could be found in a peer-reviewed source. Each evaluator first used the website UpToDate to search for a peer-reviewed source, and if it did not produce adequate results, they then used PubMed, Google Scholar or a search engine of their choice. It seems that if they could not find the fact in the chosen source, or it was in conflict with a fact found, the authors went on to assume it was an "error" in Wikipedia.
While hundreds of facts were identified in the ten articles, each team of two evaluators rarely agreed on what these facts were. Often, the number of facts identified differed by nearly 2-fold between the two evaluators. Furthermore, in 37% of cases the two evaluators did not agree on the verification of the assertion. McNemar's test was then inappropriately used to test an unclear null hypothesis. No errors in Wikipedia are mentioned directly in the original journal article. When one of us (J.H.) spoke with the lead author, he declined to release the data, stating that he may wish to publish more on the topic and wished to protect the researchers. Nevertheless, he made claims to the popular press about his findings. Some of the "errors" he mentioned, however, accurately reflected the best available peer-reviewed sources. For example, he claimed that blood pressure should only be checked twice, and not three times (as stated on Wikipedia) to make a diagnosis of hypertension. Compare this with current recommendations of the practice guidelines for managing hypertension provided by NICE (National Institute for Health and Care Excellence) [5] and cited by the relevant Wikipedia article. Given the official role of NICE in setting health policy in England and Wales, it is somewhat ironic that The Daily Telegraph, a UK paper, repeated this incorrect statement. This example shows that differences are not necessarily errors. It also shows how a sourced statement in Wikipedia may be at least as reliable as alternative statements contained in contradictory peer-reviewed sources. Instead of considering such issues, the authors merely claim that "it can be argued that these assertions on Wikipedia represent factual errors".
Whereas this study was based on assessments made by a couple of students, Wikipedia is built by a consensus of people, many of whom are experts. A recent survey of our medical editors found that 52% have either a masters, PhD, or MD, and another 33% have a BSc (unpublished data). We also work with a number of health care organizations to improve Wikipedia’s health care content including the United States National Institute of Health, the Cochrane Collaboration, and the UCSF School of Medicine.
Wikipedia has well established guidelines for what counts as a suitable medical source: we recommend the use of meta-analyses or systematic reviews published in well-respected journals from the last 3-5 years; position statements of national or internationally recognized medical bodies; or major textbooks. Is Wikipedia a perfect source? No, but other studies suggest that its quality is broadly similar to (and sometimes better than) many respected sources.[6] Hasty has created an unvalidated method to judge the quality of medical literature. He has then applied this method to a single source: Wikipedia. Even if we were to accept that getting two students to sample the literature on a topic in a haphazard way provides an appropriate basis for measurement (which we do not), without a comparator this single data point has little meaning. It would be interesting to know what their findings would have been if this method had been applied to NICE guidelines or a health information website such as eMedicine. We believe that this study should be viewed with skepticism, and that the authors should make the underlying data available to all for review.