Wikipedia talk:Identifying reliable sources (medicine)/Archive 5

This is an archive of past discussions about Wikipedia:Identifying reliable sources (medicine). Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.

Archive 1

←

Archive 3

→

Alternative medicine (again)

This edit by Razorbelle (talk · contribs) had the edit summary "(gen (copy-editing))", and largely it was. However, it introduced the following statement:

For example, discussion of unsupported medical claims must be attributed or qualified in a manner such that readers understand that the position only represents the opinion of its proponents.

into the "Summarize scientific consensus". There are several problems with this:

It is a new guideline restriction that is likely to be contentious and was introduced during a "copy-edit". Naughty. This should be discussed first.
The use of an easter egg piped link to show the author's POV that alternative medicine == unsupported medical claims. We don't that on article's never mind guidelines. No matter what our views are on alternative medicine, the link makes the fundamental mistake of assuming mainstream medicine never makes "unsupported medical claims". There's a multi-billion dollar industry that sells medicines without sufficient evidence. It even invents diseases in order to sell medicines without sufficient evidence.
IMO, the guideline achieves balance by treating all medical claims in the same way. There is no higher or lower bar for alternative medicine.

I've removed the text pending discussion. Colin°^Talk 13:30, 26 February 2011 (UTC)

I would like to respond to your comments:

Indeed, if it is a new guideline restriction, then it was proper to revert it in favor of a thorough discussion here. However, it was a good faith edit that I believed represented consensus and clarified the meaning of the sentence prior to it. It would be very helpful if others would comment as to whether it does or not. In the meantime, we should remove any controversial material from the body of the guideline.
It is not an easter egg (see: "that which has not been shown . . . to be effective" in alternative medicine).
If we are adopting the definition above, then unsupported medical claims would fall under alternative medicine. While I personally do not agree with this perspective, I was attempting to use the existing definitions in order to avoid further misunderstanding. Otherwise, I would have liked to explain that attribution or qualification must also be used when discussing the latest promises of phase II clinical trials, which may contain unsupported medical claims, but would generally not be considered alternative medicine. I understand your concerns; perhaps the sentence could be revised as to avoid an unintended point of view.

Thank you for your understanding. razorbelle (talk) 16:43, 26 February 2011 (UTC)

I'm sure it was a good faith edit and not an attempt to sneak a significant change inside an edit pretending to be trivial copy-edits. It is best to keep copy-edits from adding or removing anything significant.

The definition of alternative medicine at the WP article (medicine that hasn't been shown to work) is probably not one shared by its adherents. But the main issue is one of set equality, not containment. There exist plenty examples of unsupported medical medical claims that fall outside the area known as alternative medicine. For the weak data from early human trials, it is probably better to handle that by either showing editorial restraint or to emphasise the weakness of the evidence to date. I don't think we need to start naming the 10 people on the research paper and claiming that only those 10 currently hold the belief that they have found something useful. Colin°^Talk 20:40, 26 February 2011 (UTC)

I believe that the preceding sentence, such views must be presented in the context of their acceptance by experts in the field, covers this. I do not remember if I am responsible for that language, but I might be. Colin brings up a good point that acceptance by medical experts and well-supported by quality research and theoretical foundation are, sadly, separate matters. Do we need to tweak this on the guideline level, or is it clear enough to use wherever it comes up in the articles? - 2/0 (cont.) 20:51, 26 February 2011 (UTC)

Thank you for your advice. While I have made similar changes during copy-editing of articles in the past, I should have taken extra care given the nature of the guideline. I will be sure to make any substantive changes in independent edits in the future. razorbelle (talk) 20:59, 26 February 2011 (UTC)

Many common treatments and practices in mainstream medicine are based on conventional wisdom and habit, rather than evidence. Some examples off the top of my head:

There's no good evidence that cough syrup actually suppresses coughs, but NyQuil and Robitussin aren't generally considered "alternative medicine".
There's evidence that refusing to allow patients to eat solid foods after (non-GI-tract) surgery until audible bowel movement has resumed actually increases complications and lengthens hospital stays, but it's still a "standard operating procedure". Is that "alternative medicine" in your books?
"Standard operating procedure" requires patients to fast from all food and liquids for eight or more hours before elective surgery (to make sure that your stomach is empty, so that you're less likely to vomit under anesthesia). The gut transit time for a small glass of water (on an empty stomach) is about ten minutes, not eight hours. But we still don't call this rule "alternative medicine".
Standard chemotherapy just doesn't work in certain kinds of cancers, but when an oncologist offers it anyway, we don't call it alternative medicine.

It is a mistake to conflate "effective" with "mainstream". The best case scenario is that what is effective will eventually become mainstream, and that what is ineffective will stop being mainstream, but it's not completely true at this time. WhatamIdoing (talk) 21:18, 26 February 2011 (UTC)

I could not agree more (see: previous replies). Would we have consensus to restore the original sentence without explicitly referencing alternate medicine? razorbelle (talk) 21:36, 26 February 2011 (UTC)

I don't think it works, in the end. The statement was

For example, discussion of unsupported medical claims must be attributed or qualified in a manner such that readers understand that the position only represents the opinion of its proponents.

Who would you list as "the proponents" of cough syrup? "If you've got a cough, you might try taking some cough syrup" is the mainstream view. Wikipedia usually reserves attribution for minority views, which in this instance is the evidence-based one, rather than views that (in the opinion of the editor) are "unsupported" by evidence. WhatamIdoing (talk) 21:47, 26 February 2011 (UTC)

Agree that this doesn't work. It looks like instruction creep. II | (t - c) 22:00, 26 February 2011 (UTC)

We could put a similar point as the first sentence of the final paragraph in that section: Finally, make readers aware of any uncertainty or controversy and of the level of evidence for and acceptance of any theory or idea. Could use better wording, of course, but it segues naturally into the next section, WP:MEDASSESS. - 2/0 (cont.) 22:04, 26 February 2011 (UTC)

MEDRS or Alice in Wonderland?

An “Integrative Medicine” article abstract goes like this, (paraphrasing slightly) - “Weight reduction and exercise are known to reduce the risk of heart disease. We did a study that added 'mindfel meditation' to weight loss and exercise, and the total intervention reduced risk. This shows Integrative Medicine methods can be used to reduce risk of heart disease, possibly by incorporating weight reduction and exercise.”[1] How should MEDRS be used to deal with such absurd reasoning in medical publications, which time and again is then used to reference WP article lines like “IM methods can be used to reduce the risk of heart disease”? PPdd (talk) 00:57, 3 March 2011 (UTC)

The absurd reasoning is not found in the medical publication, but rather the original research of the editor introducing the unverified statement. razorbelle (talk) 04:05, 3 March 2011 (UTC)

No need to be snarky. PPdd actually read the study. Maybe you should too. -- Brangifer (talk) 02:56, 5 March 2011 (UTC)

I don't understand where the OR is. All that was done is remove the speculation part of the conclusion. PPdd (talk) 20:05, 3 March 2011 (UTC)

The statement is considered original research because it fails verification; it cannot be derived from the referenced document without an improper synthesis of material that advances a new position. Wikipedia editors cannot introduce a statement that explicitly contradicts the conclusions of the study. We need not invoke WP:MEDRS to remove such material. Furthermore, a single primary source cannot be introduced as authoritative. Otherwise, we could still assert that hormone replacement therapy reduces the risk of heart disease. razorbelle (talk) 03:05, 4 March 2011 (UTC)

MEDRS clearly states primary studies are to be used with caution; has this study been reproduced and evaluated by a secondary review? If not, we should avoid using it until it has been. Yobol (talk) 05:42, 3 March 2011 (UTC)

Correct on primary source, but the exact same bad reasoning could be applied to IM secondary reviews of such primary studies. The question is how to weed it out with something from MEDRS. PPdd (talk) 20:05, 3 March 2011 (UTC)

The difficulty is that if apparently professional people have published something in an apparently professional publication, then WP:V on its own won't stop us reporting it. You may feel that the conclusion is wrong but your opinion isn't worth anything. This is due to WP's publication model: that in order for us to allow anons to write our articles, we can't trust the views of these anons and must only trust other sources. Since we don't allow original research, WP cannot change the world. If the world wants to (incorrectly) believe X reduces the risk of heart disease, then we have to follow.

One solution is WP:WEIGHT. Is the view that "X can be used to reduce the risk of heart disease" one that is accepted by the majority of authorities on heart disease (e.g. professional consensus statements, national clinical guidelines, Cochrane reviews, etc). You might be able to use this to reduce or even eliminate the suggestion that it helps simply because it is an extreme minority viewpoint. But it won't be easy and you'll get a reaction that those professionals and government guidelines are in the pay of big pharma, etc. Again, that may be so but WP cannot be used to change the world.

Your best bet is a clear statement from someone like Cochrane that they have reviewed these therapies and found them wanting. It is all well and good for us to be able to review a study and spot the flaws in the methodology or conclusions, but we need other people to have done the same and published. Colin°^Talk 20:49, 3 March 2011 (UTC)

Really, this isn't surprising. Quoting (not paraphrasing): "Techniques used in implementation included mindfulness meditation, relaxation training, stress management, motivational techniques, and health education and coaching. Subjects randomized to the comparison group received usual care (UC) without access to the intervention" ... "A multidimensional intervention based on integrative medicine principles reduced risk of CHD, possibly by increasing exercise and improving weight loss". Similar trials have been done before with similar results. It's not surprising that health education and motivation would reduce cardiovascular disease. Admittedly, it's more expensive and more difficult, both for doctors and patients, than prescribing or popping a pill. II | (t - c) 21:34, 4 March 2011 (UTC)

They took the pre-existing effective treatments of "exercise and weight loss", added junk on top and called the combined thing integrative medicine, and because "exercise and weight" loss works, they concluded "integrative medicine works". That's absurd reasoning straight out of Alice in Wonderland. There must be a way to have common sense apply to keep its conclusion out of WP. PPdd (talk) 21:40, 4 March 2011 (UTC)

Dude, Wikipedia is not really the place to campaign for your views. Give it a break. II | (t - c) 00:37, 5 March 2011 (UTC)

Actually, II, the reverse would be surprising. Who'd expect that "take this prescription and see me in six months" would be as effective as "take this prescription and follow that extensive regimen under weekly medical supervision"? But as Colin says, ya gotta get a wp:MEDRS. LeadSongDog come howl! 23:32, 4 March 2011 (UTC)

Hmm - I think you and I are in agreement. I said the results are not surprising. This is logically equivalent (at least loosely) to saying that if the reverse of these results had appeared, it would be surprising. Can't really say much about the study - it appears the authors didn't track medication use in either of their groups ("it is possible that we activated patients to take their medication or talk to their doctors about further medication, and that these changes caused part of the improvement"). II | (t - c) 00:37, 5 March 2011 (UTC)

Didn't track med use? Why am I not surprised? ImperfectlyInformed, I brought it here because it is an actual example from the IM article and I didn't know how to deal with it. (I don't normally read IM journals.) I was hoping for some suggestion from MEDRS thinkers. I am not giving up that someone will think of a good way out (please don't suggest IAR). PPdd (talk) 00:46, 5 March 2011 (UTC)

Here's a joke - Department of Medicine, Duke University Medical Center, Center for Health Services Research in Primary Care, Durham VA Medical Center, Durham, NC, USA, Duke University Medical Center, Department of Biostatistics, Duke University Medical Center, Department of Community and Family Medicine, Duke University Medical Center, Department of Obstetrics and Gynecology, Duke University

Actually, that was not a joke but a copy from the abstract about the authors. LOL. :) PPdd (talk) 00:54, 5 March 2011 (UTC)

Maybe a way out is BLP. Since it is a primary source, and it makes the authors look like--- PPdd (talk) 00:59, 5 March 2011 (UTC)

Indeed. It's incredible that it got published. -- Brangifer (talk) 02:56, 5 March 2011 (UTC)

No, and your attitude suggests that you're trying to re-write the article to fit your personal point of view.

If "mindful meditation" actually helps people lose weight or get exercise, then we're okay with that. The mechanism is probably as simple as the fact that somebody who 'meditates' every single day on their need to exercise and lose weight is more likely to make healthful choices than someone who only thinks about it when he gets weighed at the doctor's office, but that doesn't matter: If the source said that the intervention group did better than the control group, then we can report that. We don't have to make it sound like they believe that meditation has magical qualities. WhatamIdoing (talk) 01:42, 5 March 2011 (UTC)

WAID, please stop implicitly acusing me of bad faith time and again. I didn't say meditation was unrelated to heart disease. I said that a study adding it to diet and exercise, already known to be effective, without comparing the combination to diet and exercise alone, then drawing conclusions about "integrative medicine" as using alt meds, is absurd, so how to deal with this absurdity in a WP:commonsense way? PPdd (talk) 02:17, 5 March 2011 (UTC)

By reporting what the source says even if you think it absurd. Really: The biggest problem with "diet and exercise" is getting people to actually do it. If waving feathers dipped in chicken blood over people's heads during a new moon gets more people to lose weight than earnest entreaties from a person sporting a white coat and a stethoscope, then bloody feathers "works". You've got a randomized controlled trial showing that "personalized health planning", including stress management techniques, works better than "usual care". So report what the source says, and quit worrying about whether the outcome seems absurd to you. We don't care what you think about the outcome. We care what type of study it was. WhatamIdoing (talk) 02:29, 5 March 2011 (UTC)

To be fair, we have a fair amount of editorial discretion. I don't think this is a good source to be citing in our articles. If you're in a dispute, you could try a WP:3O, RfC, or drop by WT:MED for help. II | (t - c) 02:31, 5 March 2011 (UTC)

OMG! I'm a bit late to this thread, and when I read this I thought PPdd was being sarcastic, but that's actually basically what it says! An equivalent situation would be to design a study of headache patients, use meditation and aspirin as treatment (thus qualifying to to be called IM), then note that the group that got the TLC in the meditation + aspirin group did slightly better than aspirin alone, then measure again later and note that the meditation + aspirin group was doing even better because, unknown to the researchers, their headaches kept coming back and they had to privately increase their dosage of aspirin, resulting in fewer headaches. Their conclusion? That combining meditation with aspirin reduces headaches, and that increasing the dosage of aspirin reduces headaches even more, thus justifying an integrative medicine approach. Can such a study be trusted? Not a bit, just like this one. It's poorly designed and poorly controlled. It's such a poor example that it can only be used as an example of poor research methods and faulty logic. -- Brangifer (talk) 02:56, 5 March 2011 (UTC)

I'm always sarcastic, or at least try to be. My citing the Duke University meta-joke, and BLP remark was sarcastic. There's probably a name for the fallacy, something along the lines of "most simplified example of Occam's Razor that is logically possible".

Aburdity on top of absurdity - Meditation probably does have a cardiovasculare effect, but the study confounded it with diet and exercise, instead of measuring it. I pray that prayer is never found to have health benifits. :) PPdd (talk) 03:41, 5 March 2011 (UTC)

So, it turns out that paper has in fact been cited a few times, including reviews in PMC 2515414 and PMC 2952316. I haven't spotted anything addressing the question though. LeadSongDog come howl! 06:01, 5 March 2011 (UTC)

I found 19 articles (reviews, dissertations and one thesis) citing this study:

Cited by (ordered by impact factor)

Lin JS, O'Connor E, Whitlock, 3 March 2011 (UTC)

No nCounseling to Promote Physical Activity and a Healthful Diet to Prevent Cardiovascular Disease in Adults: A Systematic Review for the U.S. Preventive Services Task Force. Annals of Internal Medicine. vol. 153 no. 11 736-750 (Abstract) (2008 impact factor of 16.2)

Includes Edelman et al. (2006) as an instance of high-intensity intervention (estimated >360 min) with no primary care physician role in intervention, conducted in or recruited from primary care or health plan, but does not comment specifically on this study.

Sheridan SL, Viera AJ, Krantz MJ, Ice CL, Steinman LE, Peters KE, Kopin LA, Lungelow D (2010) Cardiovascular Health Intervention Research and Translation Network Work Group on Global Coronary Heart Disease Risk. The effect of giving global coronary risk information to adults: a systematic review. Arch Intern Med. 8;170(3):230-9. Review. PMID 20142567 (Full text) (2009 impact factor: 9.813)
Described Edelman et al. (2006) as a good quality RCT which took place in a clinical setting that showed a small, statistically significant reduction in 10-year global CHD risk over 10-12 months, using a Framingham calculator. It concludes "In summary, studies showed mixed effects of global CHD risk information on predicted CHD risk, which seemed to be related to the intensity of the intervention provided. Studies with repeated global risk presentation or repeated doses of counseling showed small significant reductions in 10-year predicted CHD risk (on the order of 0.2%-2% in studies using Framingham calculators), which would be clinically important (number needed to treat, 50-500) if applied across the population of moderate- to high-risk individuals.
Kreitzer MJ, Sierpina VS, Lawson K (2008) Health coaching: innovative education and clinical programs emerging. Explore (NY). 4(2):154-5. PMID 18316060 (Impact factor: 5.65)
Cites Edelman et al. (2006) as an example of the utilization of health coaches within an integrative medicine model for a particular patient group.
Chiesa A, Serretti A (2010) A systematic review of neurobiological and clinical features of mindfulness meditations. Psychol Med. 40(8):1239-52. Epub 2009 Nov 27. Review. PMID 19941676 (2008 Impact Factor: 4.718)
Includes Edelman et al. (2006) in list of references but does not mention it in the text. Concludes: "However, given the low-quality designs of current studies it is difficult to establish whether clinical outcomes are due to specific or non-specific effects of MM."
Angermayr L, Melchart D, Linde K. Multifactorial lifestyle interventions in the primary and secondary prevention of cardiovascular disease and type 2 diabetes mellitus--a systematic review of randomized controlled trials. Ann Behav Med. 2010 Aug;40(1):49-64. Review. PMID 20652464 (5 Year Impact Factor: 4.388 - 2007)
Bundled Edelman results, no comment on the individual study.
Davis DA., Chawla NV, Christakis NA. & Barabási -L (2009). Time to CARE: a collaborative engine for practical disease prediction. Data Mining and Knowledge Discovery 20 (3): 388–415. doi:10.1007/s10618-009-0156-z. ISSN 1384-5810 (Impact Factor: 2.95 - 2009)
Cites Edelman et al. (2006) as an example of "other (proactive) interventions."
Brand CA (2008) The role of self-management in designing care for people with osteoarthritis of the hip and knee. eMedical Journal of Australia. 189 (10): S25-S28 (full text) (Impact factor of 2.894 - July 2010).
Cites Edelman et al. (2006) as example of the use of a coaching program.
Dombrowskia SU, Avenella A & Sniehott FF (2010) Behavioural Interventions for Obese Adults with Additional Risk Factors for Morbidity: Systematic Review of Effects on Behaviour, Weight and Disease Risk Factors. Obesity Facts 3:377-396 (DOI: 10.1159/000323076) (Full text) (Impact factor: 2.11)
Bundles Edelman et al. (2006) results into systematic review.
Hartog CS. (2009) Elements of effective communication--rediscoveries from homeopathy. Patient Educ Couns. 77(2):172-8. Epub 2009 Apr 15. Review. PMID 19372024. (Impact Factor: 1.975; 5-Year Impact Factor: 2.415
Cites Edelman et al. (2006) as an example of research into complementary medical care.
Khorsan R, Coulter ID, Crawford C & Hsiao A-F (2011) Systematic Review of Integrative Health Care Research: Randomized Control Trials, Clinical Controlled Trials, and Meta-Analysis. Evidence-Based Complementary and Alternative Medicine; Volume 2011 , Article ID 636134, 10 pages doi:10.1155/2011/636134 Full text (Impact Factor: 1.954)
Uses Edelman et al. (2006) to support: "Using a relationship-centered, mind-body approach (including mindfulness meditation, relaxation training, stress management, motivational techniques, and health education and coaching) in supporting behavior change, the study reported significant improvements in the 10-year cardiovascular risk as measured by the Framingham risk score (FRS) compared to usual care."
Rolley JX, Davidson PM, Dennison CR, Ong A, Everett B, Salamonson Y. (2008) Medication adherence self-report instruments: implications for practice and research. Journal of Cardiovascular Nursing 23(6):497-505. PMID 18953213 (Impact factor: 1.533)
Cites Edelman et al. (2006) as an example of integrated disease-specific outcome assessments of medication adherence.
Cohen SM; Kataoka-Yahiro M (2009) Themes in the Literature Related to Cardiovascular Disease Risk Reduction. Journal of Cardiovascular Nursing. 24 (4) 268-276 doi: 10.1097/JCN.0b013e3181a6de90 (Abstract) (Impact factor 1.533)
"In a randomized controlled study by Edelman et al, researchers found modest improvement in biological outcome measures after participants attended educational sessions supplemented by individualized coaching and telephone follow-up."
Janeway D. (2009) An integrated approach to the diagnosis and treatment of anxiety within the practice of cardiology. Cardiol Rev. 17(1):36-43. Review. PMID 19092369 (Full text) (Impact factor will be assessed in 2011)
"(Edelman et al., 2006) demonstrated that after 10 months of a personalized health plan consisting of mindfulness meditation, relaxation training, stress management, and health education and coaching, patients in the active treatment group had lowered risk, lost more weight, and exercised more frequently compared with a usual care group."
van Steenkiste B, Grol R, van der Weijden T (2008) Systematic review of implementation strategies for risk tables in the prevention of cardiovascular diseases. Vasc Health Risk Manag. 4(3):535-45. Review. PMID 18827904, PMC2515414 (Full text) (Can't find impact factor)
Cites Edelman et al. (2006) as an example of delegating to a nurse or health coach the task of involving the patient in health-management decision-making.
Erhardt L, Moller R, Puig JG (2007) Comprehensive cardiovascular risk management--what does it mean in practice? Vasc Health Risk Manag. 3(5):587-603. Review. Erratum in: Vasc Health Risk Manag. 2008;4(3):741. PMID 18078010; PMC 2291303 (Full text)
"(Edelman et al., 2006) showed that a multidimensional integrative approach, which identified specific health behaviors important for each patient to modify, was able to significantly reduce the risk of CVD. The patient, together with a health coach and a medical provider, devised a personal health plan which was driven not only by CV risk reduction but also the interests of the patient. Techniques used to help implement the health plan included meditation, relaxation training, stress management, motivational techniques, and health education and coaching."
Arvidsson D (2009) Thesis: Physical activity and energy expenditure in clinical settings using multisensor activity monitors. Full text)
Uses Edelman et al. (2006) to support: "Also, there has been a growing interest in educating clinical practitioners to coaches facilitating the patient’s behavioral change process towards a healthy lifestyle." (sic).
Spadaro KC (2008) Dissertation: Weight loss: Exploring self-regulation through Mindfulness Meditation. (Full text)
Uses Edelman et al. (2006) to support "Research studies involving lifestyle interventions that include meditation and yoga had positive results in cardiac and diabetes risk factors."
Hollenberg D & Muzzin L (2010) Epistemological challenges to integrative medicine: An anti-colonial perspective on the combination of complementary/alternative medicine with biomedicine. Health Sociology Review. 19(1) 34-56 doi: 10.5172/hesr.2010.19.1.034. (Abstract) (Impact factor 1.486 - 2009)
Uses Edelman et al. (2006) to support: "certain studies are now emerging that demonstrate a statistically significant clinical benefit of a combined IM approach."

Davis KK (2008) Dissertation: Effect of mindfulness meditation and home-based resistance exercise on weight loss, weight loss behaviors, and psychosocial correlates in overweight adults. (Full text)

Uses Edelman et al. (2006) to support: "A number of studies have shown beneficial effects of mindfulness on physical health and psychological well-being."

The only review published in a journal with a known impact factor to imply the improved risk was due to the use of an integrative medicine approach was in Evidence-Based Complementary and Alternative Medicine, the others say it is likely due to the intensity of the intervention (how often the patient is reminded of the risk associated with certain behaviours, i.e., weight gain and lack of exercise). --Anthonyhcole (talk) 15:14, 5 March 2011 (UTC)

I think this is moving away from the sort of discussion that should be on this talk page. The above (very useful) information really belongs with the article talk page. All we've done here is show how important it is to use reviews rather than primary research papers. Using research papers alone exposes us to the POV of editor cherry picking and on the ability of editors to spot flaws. Research papers alone carry no WP:WEIGHT. Colin°^Talk 16:08, 5 March 2011 (UTC)

Agree. I didn't know which article this related to. --Anthonyhcole (talk) 16:12, 5 March 2011 (UTC)

Throwing Colin's weight around is the best solution so far, but when I have tried throwing my own WP:WEIGHT around, it didn't move anybody. E.g., a "trigger point-> acupoint" ("fringe-> alt med") primary study that did not even define how big an area or volume a "point" is. With 400 acupoints, it would be unlikely none of them was close to some anatomical structures. Colin mentioned Cochrane, but their "review" of P6 acupoint and pain is preposterously sloppy, and there is a scathing blog about all of their errors by a well known MD, but their abstract does not indicate this, and the blog is NRS. Also, a bunch of homeopath/acupuncturists published their "systematic review' conclusion in an "evidence based alt med journal" (self contradictory title?) that acupoints are associated with the brain area for the organs they are supposed to go with. These and the IM example are all examples where Wikipedia's "common sense" does not seem to be working as to medical sources. PPdd (talk) 19:06, 5 March 2011 (UTC)

Can you point me to the discussion about "trigger point → acupoint" please, PPdd? --Anthonyhcole (talk) 19:18, 5 March 2011 (UTC)

here is one in the TP article, “In a June 2000 review, Chang-Zern Hong correlates the MTrP "tender points" to accupunctural "ah shi" ("Oh Yes!") points, and the "local twitch response" to acupuncture's "de qi" ("needle sensation"),[10] based on a 1977 paper by Melzack et al.[11] Peter Dorsher comments on a strong correlation between the locations of trigger points and classical acupuncture points, finding that 92% of the 255 trigger points correspond to acupuncture points, including 79.5% with similar pain indications.[12][13]”. There is even more at the acupuncture point and acupuncture articles. I tried to merge the acupuncture point article to the acupuncture so as to help keeping track of it all, but I failed to get consensus. Some discussion is here, here, and [2]. here is some more discussion. I believe you took care of the problem in the fibromyalgia article

Thanks. --Anthonyhcole (talk) 20:20, 5 March 2011 (UTC)

Could you guys please take this elsewhere. This is not WP:ALTMEDBASHING nor is it WP:LITERATUREREVIEW. Come back when you want to discuss the guideline text. Colin°^Talk 20:30, 5 March 2011 (UTC)

Sorry Colin. I was just responding to your (good) WEGHT recommendation, then Anthonycole's question. I am still wondering what to do when an apparently MEDRS source overtly uses absurd reasoning. This section is not based on an alt med journal, but a mainstream MEDRS one with apparently mainstream academic authors. The only defect per WP is that it is a primary source study, which is incidental to the abstract problem of over absurdity in medical conclusions. PPdd (talk) 20:44, 5 March 2011 (UTC)

What does "This is not WT:LITERATUREREVIEW" mean, Colin? Are you addressing me there? Are you belittling and mocking my contribution? That's what it sounds like. Could you clear that up for me, please? --Anthonyhcole (talk) 21:06, 5 March 2011 (UTC)

I don't know how you get that impression, Anthony. This talk page is for discussing changes or improvements to the guideline text. It is not a forum for debating the merits or otherwise of alt-med or the failings of specific research or reviews of that research. Colin°^Talk 23:35, 5 March 2011 (UTC)

Just me being neurotic. --Anthonyhcole (talk) 05:20, 6 March 2011 (UTC)

Is Cochrane MEDRS?

Is Cochrane MEDRS? Is The Scientific Review of Alternative Medicine MEDRS? Which analysis is more RS according to any reasonable WP:Common sense? PPdd (talk)

Reliable does not mean "always right". Look at the rubbish published in 1998 by The Lancet. If you want to discuss this particular issue (acupuncture review by Cochrane vs SRAM), please post at WT:MED or Wikipedia:Reliable sources/Noticeboard. Colin°^Talk 20:30, 5 March 2011 (UTC)

Thanks, I am fairly new (edit counts are misleadingly high) and didn't know about those places to post. Separate questions are (1) how do I establish that articles (not posts) at Science Based Medicine are MEDRS, and (2) how do I use highly reasonable posts at Scientific Review of Alternative Medicine? (Incidentally, NEJM published an outrageous conclusion about acupuncture and IM, I was in email correspondence for a while with the editor in chief about it, and he never admitted the error. So its not just Lancet.) PPdd (talk) 20:49, 5 March 2011 (UTC)

You'd want to use the current Cochrane report, not the old one, which is what was complained about.

We've been letting this slide with you, but it is technically impossible to answer the question that you've asked. Reliability involves comparing the source to the statement that you want to include in the article. So "Is this MEDRS?" is nonsensical: It's like asking "Is this more?" without indicating what it should be compared to. WhatamIdoing (talk) 21:00, 5 March 2011 (UTC)

Thanks for the cochrane update, and re MEDRS in the abstract absense a specific wording for which it supports, your nonsense comment makes sense. (There is an interesting author, Gene Wolfe, who uses "we" in the way you do in his "Knight-Wizard" series.) :) PPdd (talk) 21:48, 5 March 2011 (UTC)

The purpose of the guideline is to prevent these kinds of questions. If it continues to be ambiguous, perhaps we should revise the guideline? razorbelle (talk) 22:17, 5 March 2011 (UTC)

WAID taught me about WP:CREEP, and a revision would probably lead to it. If it comes up again re any specific MED examples, I will bring it here if it cannot be resolved at that article's talk page. PPdd (talk) 22:42, 5 March 2011 (UTC)

I hope that if you have further questions about specific sources, that you'll follow the directions at the top of this page. Look for the bit that begins "To discuss reliability of specific sources..." And when you follow those directions, please be sure to include not only the source, but the statement that the source should be supporting. WhatamIdoing (talk) 00:04, 6 March 2011 (UTC)

I didn't notice it at the top. I will go there in the future. Thanks. PPdd (talk) 00:07, 6 March 2011 (UTC)

That's important advice. To avoid instruction creep by constantly tweaking and modifying policies and guidelines, we can instead use consensus and third party opinion. That's what's good about taking medical sourcing, MEDRS questions to WT:MED. There you'll get comments from medical experts and their opinion can often help to establish a pragmatic consensus in a given situation that's tailor made to the exact source and it's proposed application. Commonsense often prevails. -- Brangifer (talk) 05:47, 6 March 2011 (UTC)

MEDASSESS

We have stated at wp:MEDASSESS that:

The best evidence comes from meta-analyses of randomized controlled trials (RCTs) and systematic reviews of bodies of literature of overall good quality and consistency addressing the specific recommendation. Narrative reviews can help establish the context of evidence quality. Roughly in descending order of quality, lower-quality evidence in medical research comes from individual RCTs, other controlled studies, quasi-experimental studies, and non-experimental studies such as comparative, correlation, and case control studies. Although expert committee reports or opinions, along with clinical experience of respected authorities, are weaker evidence than the scientific studies themselves, they often provide helpful overviews of evidence's quality. Case reports, whether in the popular press or a peer reviewed medical journal, are a form of anecdote and generally fall below the minimum requirements of reliable medical sources.

This leaves the impression (I think incorrectly) that there is an inclusion line drawn just above the threshold of case reports and below correlation studies. In the extreme, that would reduce to the absurd case of N=2 being admissible. Can we clarify this text (without getting all wp:CREEPy? LeadSongDog come howl! 17:40, 9 March 2011 (UTC)

Well, "Case reports..." could be expanded to "Case reports and small case series..." WhatamIdoing (talk) 18:03, 9 March 2011 (UTC)

The difficulty then becomes "how small is small?" I'm thinking in terms of applying the STROBE statement doi:10.1136/bmj.39335.541782.AD, so that there's an applicable academic standard at play rather than a contest of opinions between wp editors. LeadSongDog come howl! 18:32, 9 March 2011 (UTC)

The case in point is the recent discussion at talk:CCSVI re the use of a primary paper about a small genetic association study. The STREGA statement PMC 2730482, extensions to STROBE that are pertinent to genetic association studies, would help to draw the line. LeadSongDog come howl! 19:05, 9 March 2011 (UTC)

IMO review articles should be used primarily. This avoids a) either the research is too newer to have been put into proper context b) the research is not particularly notable. I would be willing to make exceptions for some notable papers in JAMA or the Lancet but these usually get a great deal of published commentary on them to balance there conclusions.Doc James (talk · contribs · email) 06:16, 10 March 2011 (UTC)

The general concept of using the best available source is not really in question. The text already makes that clear, but it also admits lower quality sources. The stated hierarchy is helpful, but it is rather vaguely stated. We should be able to clarify it.LeadSongDog come howl! 18:10, 11 March 2011 (UTC)

I'd be wary of making things less vague - I actually see that last sentence at the top as being too firm. It's also ambiguous: does this mean that if a review discusses a case report, the review's assessment shouldn't be included? I think we can all agree that review articles and secondary sources are the main sources that should be used, especially when widely available, but that shouldn't be confused with saying that other secondary sources (the introduction/discussion of research articles) or even the primary sections of articles should never be used or even "very rarely" used - which is a frequent confusion. It's also not true that case reports are always considered poor evidence - for pharmacovigilance, case reports are a core source (see [http://www.springerlink.com/content/ey6ja1k81gc0hh3p/ The use of evidence in pharmacovigilance

Case reports as the reference source for drug withdrawals]). Also, we need to keep in mind that systematic reviews and meta-analyses are mainly about specific clinical questions, narrative reviews can fairly short and surface-level, leading to articles which just summarize (or parrot) the basic conclusions of abstracts, and can also be fairly speculative and opinionated. It's better not to word guidelines too firmly, which can lead to bureaucratic wikilawyering. II | (t - c) 19:02, 11 March 2011 (UTC)

I like the resources LeadSongDog linked.

It might be worth explaining the difference between "evidence quality" and "good source". Narrative reviews, textbooks, and practice guideline are not properly 'evidence', but they are excellent sources. WhatamIdoing (talk) 20:20, 11 March 2011 (UTC)

Another MEDASSESS question

Had a question regarding this sentence: "Although expert committee reports or opinions, along with clinical experience of respected authorities, are weaker evidence than the scientific studies themselves, they often provide helpful overviews of evidence's quality." The way I'm reading it, practice guidelines by societies like American College of Cardiology and Infectious Diseases Society of America are lower in quality than all experimental studies, including individual case control studies. Is this the intent, or am I misreading? Yobol (talk) 22:30, 12 March 2011 (UTC)

If those guidelines are not based on human data, then yeah. See Evidence-based_medicine#Ranking_the_quality_of_evidence. Ex cathedra statements at the bottom. II | (t - c) 22:53, 12 March 2011 (UTC)

Except these societies base their guidelines on studies, and use the typical EBM guidelines. Yobol (talk) 22:58, 12 March 2011 (UTC)

They do now, but that doesn't mean that they've collected good information on everything. For example, several of the Advanced Cardiac Life Support therapi commonly used do not improve survival, but that wasn't included in the guideline until 2005. It's only been in the past few years that is has become really known that publication bias led to the exaggeration of antidepressant efficacy, although technically that's just distorted level I evidence. I'm sure it varies by committee and group; some are better than others. I have definitely read a few consensus statements and guidelines with Level III evidence. II | (t - c) 23:14, 12 March 2011 (UTC)

My point is current wording seems to say all guidelines (including ones based on high quality studies) are lower quality than any single experimental study. Is that intended?Yobol (talk) 01:10, 13 March 2011 (UTC)

Anyone? Buehler? Yobol (talk) 15:50, 15 March 2011 (UTC)

As I see it the solution should be simple: cite the studies that the guidelines cite. Lambanog (talk) 15:55, 15 March 2011 (UTC)

I agree it is confusing. The paragraph is a prose version of a table that appeared in earlier revisions such as this. It is trying to say the weakest "evidence" comes from mere expert opinion (i.e., not based on actual evidence, just judgement) or derived from clinical experience. It isn't saying that experts's opinions are worthless. And no, the solution is not to "cite the studies that the guidelines cite". Indeed our lead states that expert guidelines are among the best sources we can use. I've reworded the section in this edit. Colin°^Talk 16:15, 15 March 2011 (UTC)

That clears that up for me, thanks! Yobol (talk) 16:18, 15 March 2011 (UTC)

That change looks okay to me.

I think the critical point is that guidelines, narrative reviews, and textbooks are not, strictly speaking, "evidence" at all. There's no data there. They are secondary reports about other people's evidence. Consequently, they are truly excellent sources for editors to use, even though they are not, themselves, scientific evidence of anything. WhatamIdoing (talk) 16:25, 15 March 2011 (UTC)

It should be pointed out that narrative reviews and textbooks often discuss the primary sources and refer to them. Guidelines—if you look only at the complete end result of the process—do not. That end result completely on its own is 100% opinion. The discussion engaged in to arrive at the guidelines is of more value than the guidelines themselves and that discussion basically takes the form of a meta-analysis. The change made by Colin does not address the situation where a group of recent meta-analyses seem to contradict the meta-analysis used to arrive at a guideline. In that case what should be given precedence especially when respectable popular sources seem to be factoring in the newer evidence? Does dated meta-analysis provided by guideline trump newer meta-analysis recognized by respectable popular press? When there is such a schism is it really proper to cite only one side when discussing both is possible? Does that not violate WP:NPOV? Lambanog (talk) 12:10, 16 March 2011 (UTC)

I'm not sure I follow you. Are you describing a case where a guideline is now (potentialy) out of date since newer high-quality evidence has appeared? Oh to have an abundance of meta-analyses! Can you give an example of the situation you think is a problem. What issues would there be that this WP guideline gives the wrong advice on or fails to give the right advice on? You say that expert guideline conclusions are opinions. I'm not sure that is a helpful way to look at it as one could classify just about anything as an "opinion" with that rationale. And what is the conclusion of WP editors reviewing the experimental literature themselves? When chosing a source and stating a fact, we all make a judgement about the evidence. But it is best if we cite the judgement of known published experts. We must also not forget what text we are writing based on these sources, for context matters too. It is one thing to say X is an effective treatment for Y, but another to say X is recommended as a treatment for Y, and yet another to say Y is treated with X. These may all need different sources. Colin°^Talk 12:53, 16 March 2011 (UTC)

I'm in a bit of an editing situation with Yobol on coconut oil and there is direct relation to the situation with saturated fats. Looking at it I cannot help but feel it is the completion of the circle that included the trans fat debacle that the scientific consensus got people into. Lambanog (talk) 13:00, 16 March 2011 (UTC)

That's not really answering the question. This isn't a dispute-resolution forum. Is there something about this guideline that needs reworded? Colin°^Talk 16:17, 16 March 2011 (UTC)

Should it be clarified that a simple statement from a medical body should not be automatically held higher than a meta-analysis? I get the feeling that that is the reason the earlier wording was in there. Although the wording you introduced says guidelines usually have some sort of meta-analysis conducted, usually the most widely available final form is simply a series of statements. How much weight should be given the statements alone without the analysis? Lambanog (talk) 17:38, 16 March 2011 (UTC)

Naturally, it depends on the statement you're trying to make. "Doctors R Us said ___" can't really be superseded by some later publication by someone else.

When it appears that the state of knowledge is changing, it's often appropriate to provide all the pieces: the old mainstream view, the old minority view, the new evidence, the fact that the new evidence hasn't changed the mainstream view (yet), etc. WhatamIdoing (talk) 18:14, 16 March 2011 (UTC)

Conflict of Interest and Reliable Sources

There is a link at the end of the article to Conflict of Interest, but some discussion of what sources may have conflicts of interest is warranted. As a case in point, here [3]is some history on the journal "Radiation Research" which would seem to be a reliable source. A quote: "it is surprising —no, astonishing— that a leading radiation journal allows such obvious conflicts to remain unacknowledged" http://www.microwavenews.com/RR.html. In many such situations, conflict of interest becomes central to whether a source is reliable. Physicsjock (talk) 19:33, 25 March 2011 (UTC)

In the wikijargon, COI refers to editors' abuse of Wikipedia to promote the editors' own off-wiki interests. Sources need to be independent, as discussed briefly in WP:MEDRS#Use_independent_sources. WhatamIdoing (talk) 20:03, 25 March 2011 (UTC)

recentism

I changed some wording because the text stated that recentism is a policy, which it is not. Ward20 (talk) 18:29, 31 March 2011 (UTC)

Thank you. WhatamIdoing (talk) 21:13, 31 March 2011 (UTC)

Is the archive search engine malfunctioning?

Even when searching for highly prevalent words such as and and that in the search bar for the archive above, it only gives maximum 4 results. Does it work better for you? Mikael Häggström (talk) 12:55, 1 April 2011 (UTC)

Are websites with HONcode certification "reliable"?

How much reliability is there in websites that display a verifiable certification of being in compliance with the principles of the Health On the Net Foundation? INPOV, they are acceptable as medical references, but, as the Wikipedia article says, "HON does not have a means of verifying many of the principles, such as credentials (medical or otherwise) as stated on websites displaying the logo,", so the information provided by such websites should still be interpreted with some criticism. I think a note on HONcoded websites deserves a place on this project page. Mikael Häggström (talk) 13:07, 1 April 2011 (UTC)

Wow, that article's in rough shape. It needs to be updated and brought to a better standard. There don't appear to be many recent MEDRS discussing the foundation or its seal, but there are some, and they should be used. I've templated it for the project's attention. Let's get it fixed up first before we send readers to it for advice, shall we? LeadSongDog come howl! 13:55, 1 April 2011 (UTC)

Also, some improvement of the Health On the Net Foundation article would provide a better basis for this discussion as well. Mikael Häggström (talk) 16:19, 1 April 2011 (UTC)

This organization is a good idea, but since it has few teeth, we really can't take it into account regarding reliability. I'd say HON membership is neither necessary nor sufficient at this point. Ocaasi ^c 21:57, 2 April 2011 (UTC)

Suitable sources for medication indications

What do people think of drugs.com as a source for medical uses of medications. This link for example for information on metformin [4]. The info is created by the American Society of Health-System Pharmacists [5] Doc James (talk · contribs · email) 21:51, 2 April 2011 (UTC)

Here's the page disclaimer:

Drugs.com provides accurate and independent information on more than 24,000 prescription drugs, over-the-counter medicines & natural products. This material is provided for educational purposes only and is not to be used for medical advice, diagnosis or treatment. Data sources include Micromedex™ (updated Feb 3rd, 2011), Cerner Multum™ (updated Mar 18th, 2011), Wolters Kluwer™ (updated Mar 5th, 2011) and others. To view content sources and attributions, refer to our editorial policy.

And the editorial policy:

The information available on the Drugs.com website is displayed under a number of licensing agreements with various publishers. For our drug-database information, we rely on the solid reputation of our suppliers: Cerner Multum, Thomson Reuters Micromedex and Wolters Kluwer Health. Drugs.com does not alter the drug information supplied by these companies. However, we acknowledge that errors and omissions in drug information may occasionally occur. As our mission is to provide the most accurate, up-to-date drug information on the Internet, we ask our visitors to alert us immediately to any errors in content, including incorrect or misleading statements, by contacting us here. We aim to correct site-content errors within 48 hours. (full:http://www.drugs.com/support/editorial_policy.html)

Ocaasi ^c 22:00, 2 April 2011 (UTC)

Yes it contains content from many different organizations. The AHFS pages however are very well referenced. Should it be added as a possible reliable secondary / tertiary source for medication information? Better than emedicine IMO as reffed and less ads... Doc James (talk · contribs · email) 22:11, 2 April 2011 (UTC)

I'm not sure that I'd recommend it or disrecommend it. It's okay for many purposes, but I wonder occasionally if it has an American bias (e.g., listing only uses that the US FDA approves of). Also, it's not necessarily the best source for recent information or disputed uses. WhatamIdoing (talk) 01:13, 3 April 2011 (UTC)

The vital thing is that our sections on drug usage (whether the treatment section of a disease article or the indications section of a drug article) should be based on actual prescribing guidelines, not on research or purely academic reviews and analysis of that research. We need to make it clear to our readers what the drug is actually used for in practice, rather than jumble up all the experimental therapies or older practice. Where recommendations are described, these should ideally be based on an authority rather than just the authors of some paper we found on PubMed. And we very much need to avoid writing our own clinical guidelines. An example of getting this wrong is metformin.

I think guidelines like those produced by SIGN and NICE should be a first port-of-call. They often contain extensive discussion of the research and evidence that would be useful elsewhere in an article. Alternatives would be guidelines produced by some medical organisation concerned with a particular condition. A useful base would also be the prescribing information in databases like Drugs.com and the BNF. Perhaps folk here have ideas for other resources too.

If we put a drug article up at FAC, a typical question might be "What makes XXX a reliable source for Y". The above describes Drugs.com's own editorial policy, which seems very good. But can someone find an independent comment on Drugs.com and whether it is respected as a source of reliable, accurate and up-to-date information. Colin°^Talk 08:22, 3 April 2011 (UTC)

FDA: How can I stay better informed about drugs? Is there a reliable website FDA recommends? Try the websites www.drugs.com/fda-consumer/ or MedlinePlus. Drugs.com is designed for both consumers and health professionals. The site features FDA Consumer Update articles, videos, and slideshows. Site users can get valuable access to information about new drug approvals, use a drug interaction checker to check the compatibility of using two drugs at the same time, identify drugs from their tablet and capsule markings, ask questions to other site users, join support groups, and much more. Site users can also subscribe to the Drugs.com electronic newsletter and also sign up to receive e-mail alerts from FDA's MedWatch.

Press release: For Immediate Release: May 26, 2010. FDA Announces Collaboration with Drugs.com. Goal is to expand online distribution of Agency’s consumer health information The U.S. Food and Drug Administration announced today that it will collaborate with the Web site Drugs.comExit Disclaimer to expand access to the FDA’s consumer health information. Drugs.com seeks to provide patients with information to better manage their own health care and to assist in the reduction of medication errors. It attracts more than 12 million unique visitors each month. "The FDA’s partnership with Drugs.com means that reliable, useful, and timely health information will be available to an even wider audience,” said Beth Martino, the FDA’s associate commissioner for external affairs. “Partnerships like this are an important part of the FDA’s effort to ensure the public has easy access to reliable, useful information that can help people protect and improve their health.” The FDA's partnership with Drugs.com will provide consumers with a joint resource on the Drugs.com site featuring FDA Consumer Update articles, videos, and slideshows. The partnership will also provide access to FDA health information on Drugs.com’s mobile phone platform. "We are very excited about partnering with the FDA to provide consumers with public health and safety information on our site as well as our mobile phone platform," said Philip Thornton, CEO of Drugs.com. The complete terms and components of the partnership with Drugs.com are described in a Memorandum of Understanding published in the May 26, 2010, Federal Register and available online at http://edocket.access.gpo.gov/2010/pdf/2010-12638.pdf. Ocaasi ^c 08:40, 3 April 2011 (UTC)

Surely, with many aspects given above, drugs.com not perfect, but I lean towards saying that it's a reliable source. It's disclaimer is not more disclaiming than the Wikipedia:Medical disclaimer. Still, when the equivalent information can be found in more established guidelines, such as by SIGN or NICE, then those should be used. But my general impression of drugs.com is that it's an acceptable reference for Wikipedia entries. Mikael Häggström (talk) 15:54, 3 April 2011 (UTC)

I agree that it is a reliable source of information. I would not, however, wish to anoint it as a preferred source of information. WhatamIdoing (talk) 16:54, 3 April 2011 (UTC)

Well "preferred" is a relative concept. I'd say it was preferred to editors doing an amateur version of NICE themselves. Colin°^Talk 18:28, 3 April 2011 (UTC)

We should make recommendations for possible sources to use for drug articles. When it comes to diseases you can find a great recent review and the reviews coverage is similar to that of a wiki article. The same cannot be found for medications. There are no review that give us the information that should be in a drug article ( not that I have found anyway ). Also NICE guidelines cover diseases rather the medications. So yes a single NICE article could support a single use and thus could and should be used in that manner it is not as easy because it does not give an overview of the drug as a whole. So as Colin says this is a way to guide editors away from doing there own review of the literature. I have used the drugs.com source to rewrite the methotrexate medical uses section. Are there other sources that people know of that should be used preferential?Doc James (talk · contribs · email) 19:22, 3 April 2011 (UTC)

I don't know if it's preferential, but I sometimes use the Swedish official drug catalog (Fass.se). Most people need Google translate to understand it, which may make it a bit inconvenient, but at least verifiable. Mikael Häggström (talk) 04:57, 4 April 2011 (UTC)

Is WebMD a reliable medical source?

Should WebMD continue to be supported in the body of the guideline in light of this article featured in The New York Times? razorbelle (talk) 16:19, 3 April 2011 (UTC)

It probably still qualifies as "reliable", but I'm not sure that we want to be recommending it by name. WhatamIdoing (talk) 16:58, 3 April 2011 (UTC)

Agree with WhatamIdoing. Mikael Häggström (talk) 17:19, 3 April 2011 (UTC)

Agree it is okay but not preferred.Doc James (talk · contribs · email) 05:18, 4 April 2011 (UTC)

MEDRS and biographies

Discussion moved to Talk:Weston Price . Request for help posted at WT:MED. Colin°^Talk 07:43, 12 April 2011 (UTC)

Which RSMED "reliable source" to believe? Hmmm?

Blog about saturated fat and heart disease.

For all my medical career it's a well-known fact that it's a well-know fact that saturated fat intake contributes to atheromatous cardiovascular disease. Every major medical body has said this for 30 years and more. Of course, 30 years before that (taking us back to 1950) there was no official position on diet and coronary disease, until 1953 when Ancel Keys published his 6 country study. A timeline is here, for those of you who didn't live through a lot of this: [6] from 1956 until the official US government guidelines on fat intake were codified, saturated fat was the badguy. McGovern's Senate Select Committee issues the final version of the Dietary Guidelines for Americans in 1977, in which animal fat is the primary Bad Thing. The evidence: a bunch of epidemiology in which a lot of societies that had the "wrong numbers" (Polynesians who live on coconut and French who eat butter and fois gras) were selectively ignored. The animal studies were pitiful, and generally malnurished a bunch of animals with hydrogenated oils and no EFAs. Nobody cared. The reliable government sources considered this data reliable, due to their previous bias, due to Keys.

Okay, fast-forward to last year:

Am J Clin Nutr. 2010 Mar;91(3):535-46. Epub 2010 Jan 13.

Meta-analysis of prospective cohort studies evaluating the association of saturated fat with cardiovascular disease.

Siri-Tarino PW, Sun Q, Hu FB, Krauss RM. Children's Hospital, Oakland Research Institute Oakland, CA, USA.

Comment in:

Am J Clin Nutr. 2010 Mar;91(3):497-9.
Am J Clin Nutr. 2010 Aug;92(2):459-60; author reply 460-1.
Am J Clin Nutr. 2010 Aug;92(2):458-9; author reply 459.

BACKGROUND: A reduction in dietary saturated fat has generally been thought to improve cardiovascular health. OBJECTIVE: The objective of this meta-analysis was to summarize the evidence related to the association of dietary saturated fat with risk of coronary heart disease (CHD), stroke, and cardiovascular disease (CVD; CHD inclusive of stroke) in prospective epidemiologic studies. DESIGN: Twenty-one studies identified by searching MEDLINE and EMBASE databases and secondary referencing qualified for inclusion in this study. A random-effects model was used to derive composite relative risk estimates for CHD, stroke, and CVD. RESULTS: During 5-23 y of follow-up of 347,747 subjects, 11,006 developed CHD or stroke. Intake of saturated fat was not associated with an increased risk of CHD, stroke, or CVD. The pooled relative risk estimates that compared extreme quantiles of saturated fat intake were 1.07 (95% CI: 0.96, 1.19; P = 0.22) for CHD, 0.81 (95% CI: 0.62, 1.05; P = 0.11) for stroke, and 1.00 (95% CI: 0.89, 1.11; P = 0.95) for CVD. Consideration of age, sex, and study quality did not change the results. CONCLUSIONS: A meta-analysis of prospective epidemiologic studies showed that there is no significant evidence for concluding that dietary saturated fat is associated with an increased risk of CHD or CVD. More data are needed to elucidate whether CVD risks are likely to be influenced by the specific nutrients used to replace saturated fat.

That's a GIGANTIC meta analysis. Now, the "reliable sources" told us one thing in 1953, something else quite different in 1977, and now we're back again to 1953. How can that not make you cynical? The 4 food groups were killed by the low fat people. The 4 food groups had no backing anyway. But neither did the low fat food pyramid. This is fail after fail after fail.

And all this cynicism is not just that I'm a medically unsophisticated person following along in the newspapers. I've been deep in the medical journals following the mechanisms and formal recomendations for years. And recognizing their bias. But unable to do very much about it. Sometimes they even suceeded in fooling me: I can't know everything and my instincts are not always good. But I was ahead of a lot of THIS game.

So, watch that the same kind of thing doesn't happen to you, in any field in medicine where there are no randomized prospective controlled primary prevention studies. (Ornish isn't one). You can get major egg on your professional face this way. S B H arris 06:48, 15 April 2011 (UTC)

The only relevant aspect to the above blog on staturated fat and heart disease is that science is an ongoing process and the things that are considered "facts" may change. That WP is less concerned with "Truth" than what the consensus-accepted-version-of-reality is, means this shouldn't trouble us. We should use the best up-to-date sources we have and reflect the current expert opinion. We can't be wrong any more than a Reuters reporter is wrong when he faithfully repeats the rubbish spoken by a politician. Colin°^Talk 07:53, 15 April 2011 (UTC)

WP shouldn't be repeating the rubbish spoken by a polician as fact (except that they said it), merely because it's been run through Reuters. That's a major problem in WP:IRS. And you're wrong about science if you mean what I think you do.

And what has been labeled a "blog" above makes more or less that point, and you seem not to want to address it.

Sorry, but you can't dismiss the problem that easily. The RSMED guideline is certainly concerned (at least in part) with "truth," just as IRS is. The words "fact" and "accuracy" in WP:IRS imply correspondance with objective truth, not some socially-accepted social-text version of consensus-truth, based badly on poor grades of evidence, but accepted because authorities want to believe it. "Accurately reflect current medical knowledge" (used in RSMED) does not mean a thing, if "current knowledge" itself is held to have no permanent correspondance to reality, and is totally unstable, rather than only a bit unstable at the edges.

Let me give you an analogy with physics: the best "knowledge" in physics (conservation laws, etc) is expected to be permanent because it's so reliably predictive over the range of phenomena it was invented to describe. Advances come in extreme situations (extreme physical conditions) where the new theory is better, but the old theory is not totally wrong-- rather it still remains close to true (effectively true) in the limiting case (as with Newton versus Einstein; Maxwell vs. Dirac). In engineering, just because the theory gets better, doesn't mean the bridges now fall down. Rather, in engineering, the new theory is required to explain WHY the bridges still stay up (which is the objective fact). That is how "truth" works in science, and is how it ought to work in an encyclopedia and medicine.

In medicine, there are things that correspond with the conservation laws of physics: they are the basic truths of physiology, such as the idea that the blood carries oxygen to tissues, that aren't going to change. We also know that microbes are responsible for epidemics of transmissible diseases (not miasmas or ghosts or putrifying cadaveric particles). And in 50 years, smoking per se will still be bad for you.

The difference between medicine and the older sciences is that in medicine we have a whole lot of other junk that we put on par with the badness of smoking (or hypertension), that aren't supported by anything like the quality of data. But, due to the authoritarian nature of medicine, we have treated them as though they were. I used saturated fat intake as an simple example: We don't have any "smoker country paradox" where the inhabitants smoke like mad all the time, and get no more lung cancer or lung disease than anywhere else. Nor countries where everybody has hypertention but nothing happens. We do have that for saturated fat and have had it for years, but it was *ignored.* Likewise, if LDL cholesterol were a major independant cause of atherosclerosis over the range mildy over average, then lowering it by any means would have a major impact on the disease, and not just if it's done via statin-use (which has only a modest impact, at that). But that wasn't true, either. So, the medical establishment has been lying to itself and ignoring the quality of its evidence, in direct contradiction to the guidelines of RSMED (we now reach my point).

To sum this point up, RSMED would have WP writers partly follow the guidelines of "evidence based medicine" (except where it doesn't and instead follows authority, I suppose). But as we all know, orthodox medicine isn't even THAT careful, and runs nearly everything through the authority-filter, last. The consensus-statements of the NIH (example [7]) turn out to be no better than those from Wikipedia.

Basically, this comes down to a two-part question: 1) is WP going to be concerned with having truth (accuracy, objective fact) in their encyclopedia, or are they merely going to worry ONLY about accurately reflecting the opinions of official panels of experts, chosen by other official panels? 2) is WP:RSMED doing to try to do better? If not the last, all the evidence-based stuff in RSMED may as well go, since other people are doing that job FOR us on WP (the expert panels), and we're not savvy enough to do any of that on our own, as writers. For example, if we find that the "experts" are not following evidence-based procedures themselves, we're forbidden to point that out on our own. And if only a minority of people in health care "notice" that this emperor has no clothes (perhaps they write letters to the journal editor in protest, which are printed), we can't give their minority opinons undue weight, until they finally win out over the various biases built in to medical practice (example, drug advertising and pharma-funded studies).

So, which is it to be? Are you doing to mirror authority, or attempt to use your own judgement per evidence-based guidelines? If the first, then delete RSMED, since the parts of it that differ from WP:IRS have no function, and no justification in your WP philosophy. Admit it. S B H arris 17:13, 15 April 2011 (UTC)

I skimmed, to be honest, but I think you're missing the point. No one is claiming that the consensus of the medical literature is always right. But the consensus of the medical literature should always be the POV of Wikipedia. Perhaps I have evidence that low-dose radiation can increase lifespan by 50 years. That's great, and I should pursue the proof with vigour. But until my findings are accepted by the medical community, they're not going to be accepted here, however true or evidence-based they are. We're not a research institute or a think tank or a bunch of sages who impose our "truth" on the world. Wikipedia is, or should be, a mirror of the reliable sources, even when an individual believes those sources to be wrong and yes, even when those sources are wrong. Keepcalmandcarryon (talk) 18:17, 15 April 2011 (UTC)

I pick "mirror authority", hands down, every time. The primary alternative is that every patient cherry-picks the sources that validate his or her own personal belief, and we pretend that they're all equally reliable sources, just like the opinions of two politicians might be equally valid ways of considering a social situation.

The function of our explanation here is to help people figure out whether a published source is likely a reliable one (for certain scientific or technical claims, as opposed to which politician said something). In medicine-related articles, editors are often faced with sources that do not agree perfectly. If given a choice between an article in Journal of Medical Impressiveness that says "10%" and an article in Journal of Academic Importance that says "12%", we are trying to equip editors with the tools that help them decide when to choose "10%", when to choose "12%", when to choose "10 to 12%", and when to keep looking for other sources. WP:RS does nothing like this, because this kind of problem so rarely appears when talking about which politician said what or what the population of a city is. WhatamIdoing (talk) 18:55, 15 April 2011 (UTC)

Sbharris, please stick to the point and try to keep it short. Only a small part of MEDRS (Assess evidence quality) offers guidance for WP editors for knowing what is strong and weak evidence. You are right that it is not our job to judge the evidence and decide based on primary data what is correct or what deserves weight, which is why we advise against using primary research as sources. We should, wherever possible, be building upon the analysis and writings of experts and the weight given by others in reliable sources. The ability to recognise what makes strong evidence, to know what forms a good scientific experiment, what biases might be present in a study, are all very helpful background information that will be very helpful for many of our editors. Such a background, which our expert editors already have, greatly helps in making best use of the right sources and interpreting them correctly. Editors may come to this guideline without any background in medical research (conducting or reading) but perhaps having read something in the newspapers. As WhatamIdoing notes, it isn't always easy to judge things without these tools. You can recognise that a major meta-analysis is probably an important finding and a small case study is not, but both might be reported by the Daily Mail with equal boasting.

WP is not concerned with Truth. We have a policy that makes that clear. Accuracy is important and of course an encyclopaedia is concerned with facts. I think you're mistaken about what "truth" and "facts" are but have absolutely no desire to get into a philosophical debate about it.

You missed the point about Reuters, which is exactly that all Reuters can do is accurately report that a politician said X. It is not their job (unlike a good journalist) to investigate if X is correct and it is not their job (like a bad journalist) to impose their own political slant on the reporting. Similarly, we report what the best sources say about X. If they are wrong about X then we are wrong about X. Colin°^Talk 19:22, 15 April 2011 (UTC)

I didn't miss the Reuters point. WP:IRS uses newspapers, and regards them as reliable, for many other purposes than reporting the bald facts of a what a specific named person claims. The problems come in when they go farther, as when a reporter uses a bad and unnamed source for an important New York Times story and we (the reader) know nothing of the background. Terrible problems have resulted from this, even perhaps wars. Neither the public nor Wikipdia have learned anything about the reliability of newspapers and journalists, however. [8]

As for "WP is not concerned with Truth. We have a policy that makes that clear," you're quite simply wrong about that. [9]. You're not alone in not wanting a debate, however, because WP's policy is unclear and very often misunderstood to "clearly mean" what it does not actually say. As you demonstrated. S B H arris 19:44, 15 April 2011 (UTC)

You still miss the point about Reuters. I guess you don't know what they do. They aren't "newspapers". Oh, and you are still wrong about WP not being about Truth. You were wrong in the link you posted too. This debate is tiresome. Yawn. Colin°^Talk 14:13, 17 April 2011 (UTC)

The neutral point of view policy is designed to ensure that Wikipedia does not overstate a certain position in a scientific community, and gives due weight to "significant minorities". So, regardless of what happens in practice (and there are a lot of editors who do not appreciate NPOV, in my opinion - particularly Keepcalmandcarryon above), the saturated fat article should always have acknowledged that there was a debate within the medical community on this issue, as there is currently. There were, and are, plenty of articles discussing the anomalies. It's still controversial. As always, it's the job of more experienced editors to try to rein in people who are using Wikipedia as a soapbox - regardless of whether they're soapboxing for the "mainstream" view or the "fringe" view. There's a healthy balance which can be difficult to strike, since most people are uncomfortable with measured statements and uncertainty. Eubulides was an editor who I felt worked within NPOV pretty well, but most other people I encounter around here don't strike me in the same way. II | (t - c) 20:53, 15 April 2011 (UTC)

While I myself think that some type of "NPOV" summary is unavoidable (meaning that the editor presents the POVs but doesn't take one himself), if you want to fairly represent academic thoiught (just as in teaching an academic course) in practice the identification of what are the major views of reliable sources still comes down to guaging what are the reliable sources in the first place. And if you refuse to be binary, then doing some kind of integration of HOW reliable the source is, multiplied by how PREVALENT its position is. Even if you have that information for that (you almost never do) and some algorithm as to how to make the tradeoff (there isn't one), you still need to do the source-reliablity work BEFORE you start. That is, unless you're willing to let the sources themselves judge the reliability of sources (something I see all the time). But then you end up trying to judge the reliabity of the sources that judge the reliablity of sources. Again no escape from personal responsibility in judging bullcrap, save self-deception.

At some point, many steps removed you might think that you're now on automatic and not doing any internal truth-detection, but try summarizing a subject you don't understand sometime (Lie algebra, anybody?) and you find that you're just outputting garbage. We have a feel for that in mathematics, music, and spoken languages that we don't understand, but it's just as true for statistical inference (the danger in popular subjects like medicine being that some people think they understand what they actally do not-- fewer people make that mistake in math or music). So back to the saturate fat thing-- I'm not arguing that we shouldn't have had a "minority report" from the beginning (at least until the risk was as clear as for smoking and hypertension), but rather that the way WP actually works, precludes that very often, from happening.

Many opinions need to be held more lightly than they are, and WP needs to reflect that, somehow. However, it is unfortunately the case that the purpose of concensus reviews in science is to try to force consensus where there isn't one already (otherwise, what's the point?). And THOSE "consensus" reviews get taken at their word, except by the very few. So, each time you review you end up then with a gray-scale picture being run through a contrast filter until it's black-and-white, as happens with photocopies of photocopies, and the farther you go from the primary data, the more that's likely to happen. It happens at every level of analysis and summary, and WP is the last one! S B H arris 21:56, 15 April 2011 (UTC)

Can a primary source be used to debunk reliable secondary sources?

For the last couple of days a new editor has been arguing to include mention that a primary study claims its findings "indicate" that mercury in hepatitis B vaccines "may" cause autism, in Causes of autism. I and many others have pointed out that med articles are based on secondary sources. The new editor is pointing to this guideline saying it permits the use of a primary source to contradict or debunk secondary sources if the primary source expressly makes the debunking or contradictory claim. The relevant passage is:

Individual primary sources should not be cited or juxtaposed so as to "debunk" or contradict the conclusions of reliable secondary sources, unless the primary source itself directly makes such a claim.

Does this wording reflect actual practice on med articles? I can imagine a situation where it might apply, when a new, uncontroversial line of research challenges established consensus but hasn't yet been reviewed; but not really. Even there surely it would be prudent to wait for expert review before reporting it here. Can anybody clarify this for me? --Anthonyhcole (talk) 19:31, 22 April 2011 (UTC)

I've always taken it to mean that you can't juxtapose primary with secondary sources unless the primary source specifically mentions/compares their results in the context of the specific secondary sources used in the article. The chances of that happening are quite low, and does not apply to the cases the IP editor in Causes of autism anyways as the primary studies they want to add do not in fact mention the secondary sources in our article. Of note, the IP editor is also making an unwarranted assumption, that if the primary source does directly contradict a secondary source, then it must be included, which is also incorrect. There could also be very good reasons (WP:UNDUE being the top of the list) not to include recent primary articles even if it does not get prohibited by this clause. Yobol (talk) 19:40, 22 April 2011 (UTC)

I agree, WP:UNDUE applies here, but does that quoted section need clarifying to reflect what you've just said? I bring this up because I'm sure I saw exactly this argument being put a year ago on Talk:Autism, and the IP went on for weeks saying, "but MEDRS says I'm allowed to." If this guideline can do without "...unless the primary source itself directly makes such a claim" or if it can be clarified, it might save a bit of angst. --Anthonyhcole (talk) 19:50, 22 April 2011 (UTC)

We had an argument like this regarding a claim made in the Weston Price biography. The issue was "(i)n the 1930s, editorials and research refuted the theory of focal infection" citing 2002 Ingle's Endodontics 5th edition as a reference. The problem was that claim could be shown to be UNTRUE with the primary sources like the Southern California State Dental Association Journal (1952) and Galloway, Thomas C. M.D. (1957) "Relation of Tonsillectomy and Adenoidectomy to Poliomtyelitis" JAMA. 1957;163(7):519-521. doi: 10.1001/jama.1957.02970420001001. Even worst were there were secondary sources that showed that in this case Ingles was talking nonsense:

Bergenholtz, Gunnar; Preben Hørsted-Bindslev, Claes Reit putlich (2009) Textbook of Endodontology Wiley; page 136

Fowler, Edward B (2001) "Periodontal disease and its association with systemic disease" Military Medicine (Jan 2001)

Garg, Nisha; Amit Garg (2007) Textbook of endodontics Jaypee Brothers Medical Publishers pg 2

Saraf (2006) Textbook of Oral Pathology Jaypee Brothers Medical Publishers pg 188

Silverman, Sol; Lewis R. Eversole, Edmond L. Truelove (2002) Essentials of oral medicine PMPH usa/BC Decker; Page 159

And yet editors forgot that "The reliability of a source depends on context. Each source must be carefully weighed to judge whether it is reliable for the statement being made and is the best such source for that context." The key here is the "reliable for the statement being made" and in this case Ingle was flat out unrealible for the statement being made as it could be shown via primary AND secondary sources that focal infection theory continued well past the 1930s. This is one of the limited cases where a primary source CAN override a secondary source it shows the statement to be blatantly inaccurate.--BruceGrubb (talk) 04:39, 27 June 2011 (UTC)

Normally, if you've got a conflict between (apparently) good secondary sources, you should be citing the secondary sources for both "pro" and "con" arguments. You shouldn't cite an old primary source when you've got a newer and better secondary source that says the same thing. Consequently, in that situation, you should still be following the rule about not using a primary source to debunk a secondary source. WhatamIdoing (talk) 05:07, 27 June 2011 (UTC)

Appropriateness of a source

The second sentence in WP:V runs, "...all material added to articles must be attributable to a reliable, published source appropriate for the content in question" (emphasis mine).

It seems to me that this question of appropriateness is the main point behind much of MEDRS. A human interest story or a press release about a charity fundraiser might be more or less "reliable" for its contents, but it is not an appropriate source for statistics. I wonder whether it would be helpful to address that point more directly on this page. What do you think? WhatamIdoing (talk) 21:23, 12 May 2011 (UTC)

Contradicts itself and WP:RS

The guideline as written currently contradicts both itself and WP:RS by stating "All Wikipedia articles should be based on reliable, published secondary sources". WP:RS states "Wikipedia articles should be based mainly on reliable secondary sources". As worded this guideline is effectively excluding primary sources, contrary to policy. It's also contrary to itself, where it goes on to discuss legitimate uses of primary sources. I added the word "primarily" (paraphrasing the policy) and it's been objected to. I don't see why. --Icerat (talk) 01:52, 28 May 2011 (UTC)

Since there's been no comment, if there's no objection I'm going to edit the guideline and insert mainly - "All Wikipedia articles should be based mainly on reliable, published secondary sources" as per WP:RS policy.

Give people at least a week to comment. At WP:MED we require higher quality sourcing for medical content. --Doc James (talk · contribs · email) 15:21, 31 May 2011 (UTC)

There's no contradiction. "Based on" does not mean "uses exclusively". WhatamIdoing (talk) 15:58, 31 May 2011 (UTC)

It's not okay for this guideline to have higher standards than the policy. If the guideline is to be kept as is, then we have to make appropriate changes in the Policy. Also, if this guideline comments on universal Wikipedia procedures, such as "All Wikipedia articles..." it has to reflect the policy and common practices. We can definitely clarify whether 'based on' means exclusive or not, and make the language precise and unambiguous. Ocaasi ^c 17:01, 31 May 2011 (UTC)

I don't believe that the current wording actually represents a higher standard than the policy. WhatamIdoing (talk) 18:09, 31 May 2011 (UTC)

I agree. I don't really see a conflict here, and I don't think this guideline actually proposes a higher standard than WP:V. It's intended to provide guidance about what sorts of sources might satisfy WP:V when it comes to medical content. I don't have a problem with including "mainly", because I think that's already assumed by the existing wording, as WhatamIdoing pointed out above. Medical articles should be based (mainly) on reliable, secondary sources, but we can of course judiciously include primary sources, subject to certain strictures that are detailed elsewhere. MastCell ^Talk 18:53, 31 May 2011 (UTC)

Two points: 1) Both WP:RS and WP:MEDRS are content guidelines; WP:RS is certainly not a policy (as the original poster seems to suggest), so there can be no discrepancy between "guideline" and "policy" when WP:RS is not policy. 2) WP:OR, which is policy, clearly states "Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources." As such, this guideline (MEDRS) is clearly in line with policy, and no further action is needed. (Suggestion that the current wording excludes all non secondary sources is clearly dubious given the fact the very next sentence explains when primary sources could be used.) Yobol (talk) 18:59, 31 May 2011 (UTC)

I suppose the phrase 'based on' is misleading some. WP:PRIMARY says: "Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources. Secondary or tertiary sources are needed to establish the topic's notability and to avoid novel interpretations of primary sources, though primary sources are permitted if used carefully. All interpretive claims, analyses, or synthetic claims about primary sources must be referenced to a secondary source, rather than original analysis of the primary-source material by Wikipedia editors. Appropriate sourcing can be a complicated issue, and these are general rules. Deciding whether primary, secondary or tertiary sources are appropriate on any given occasion is a matter of common sense and good editorial judgment, and should be discussed on article talk pages." It obviously implies that based on does not mean exclusively based on. Perhaps we should make that more explicit. Ocaasi ^c 19:10, 31 May 2011 (UTC)

I guess I don't see how it can be misleading in the context of the entire paragraph. In fact, I don't see how "based on" suddenly becomes "based exclusively on". When I say "a good education is based on strong reading and math skills" I don't mean study only mathematics and literature to the exclusion of science, history, music, etc, nor would I think any reasonable person expect it to. In any event, the wording of this guideline is in line with policy; if there is need for clarification, it should probably be taken up at WP:OR first. Yobol (talk) 19:17, 31 May 2011 (UTC)

We can leave it and wait to see if others find it misleading. Maybe MEDRS is missing the second part in proximity, where it says "primary sources can be used under x, y, z, circumstances". Ocaasi ^c 19:25, 31 May 2011 (UTC)

Oh. " All Wikipedia articles should be based on reliable, published secondary sources. Reliable primary sources may occasionally be used with care as an adjunct to the secondary literature, but there remains potential for misuse. For that reason, edits that rely on primary sources should only describe the conclusions of the source, and should describe these findings clearly so the edit can be checked by editors with no specialist knowledge. In particular, this description should follow closely to the interpretation of the data given by the authors or by other reliable secondary sources. Primary sources should not be cited in support of a conclusion that is not clearly made by the authors or by reliable secondary sources, as defined above..." Nevermind. Ocaasi ^c 19:31, 31 May 2011 (UTC)

Material based on primary sources: just delete?

The guidelines don't appear to address what should be done when primary sources are cited. Should claims based on primary sources simply be removed?

I've noticed a few deletions where the material wasn't wrong but a better source would have been suitable. Is deletion the appropriate action, or is there a suitable template (perhaps something like {{better source}}). I am concerned that if we remove all material based on primary sources, a lot of good content will be lost. Either way, a clear recommendation would be useful here. pgr94 (talk) 23:44, 1 June 2011 (UTC)

I see a lot of inappropriate health claims sourced to primary research and when one looks at review articles find that either the primary research has been cheery picked or does not give the entire picture. IMO most primary research studies are not notable (there are a few exceptions). Doc James (talk · contribs · email) 23:47, 1 June 2011 (UTC)

Ultimately, I think we all want the same thing: high quality articles based on secondary sources. It's just a question of how we get there. I understand that there is a risk of cherry-picking information from primary sources. And I appreciate that this is medicine where accuracy is particularly significant. At the same time, these are peer-reviewed publications in (usually) respectable journals. Just deleting seems rather drastic. If we were take all medicine articles and delete all except claims based on review articles, it would be a huge cull. There'd be just a small fraction of material left and coverage would be extremely patchy. Unless a statement is blatantly wrong, I am inclined to think it is more constructive to keep material and encourage better sources with a "better sources needed" template. pgr94 (talk) 12:16, 2 June 2011 (UTC)

There is a range of appropriate measures keeping in mind that extraordinary content requires extraordinary sourcing. This is a great example Medicinal mushrooms of lack of enforcement. We basically get an unbalanced and mostly useless collection of information promoting a particular POV. I gave up on this page and have only watched it expand.Doc James (talk · contribs · email) 13:14, 2 June 2011 (UTC)

And then we have the primary research on animals and tissue culture / cell lines. This stuff should just be deleted outright. We have this quote "Scleroderma citrinum (common earthball) - Activity against HSV-1 and Mycobacterium tuberculosis.[101] " PMID:12865983 Which is misrepresenting the research as they did not test the mushroom but an chemical isolated from it. This sort of stuff fails notability.Doc James (talk · contribs · email) 13:16, 2 June 2011 (UTC)

I too have largely given up hope at Medicinal mushrooms. At this point an interwiki move to wikiversity might be the best approach. There's a whack of content, but it is overwhelmingly primary source based. Not that it needs to be, but the owner can't/won't/doesn't get it.LeadSongDog come howl! 16:29, 2 June 2011 (UTC)

I don't think that high-quality primary sources should always be removed, and I agree that the best first step is normally to tag them with {{primary-source inline}}.

One thing that's important is the subject of the article. A main human-oriented overview topic like Cancer should have almost zero primary sources, and basically nothing from preclinical studies. A more specific disease article might properly cite some preclinical studies, e.g., in a ==Research directions== or ==History== sections, or perhaps in a path section.

But an article about medications (or potential medications) that are currently being researched might well have nothing except preclinical studies. For example, every few years, some plant or food is big in the new about how it's going to cure all the world's ills. These are commonly based on one in vitro study. It is reasonable for us to describe the study that makes the headlines, even though it may be years before they have results from any humans trials.

On another point, if the work involves a single chemical extracted from a mushroom, plant, or living creature, it's perfectly fine to describe the research being done on it. Penicillium ought to mention penicillin, and Willow ought to mention aspirin. Similarly, it's not unreasonable to include information about the medicinal use of mushroom extracts in Medicinal mushrooms. WhatamIdoing (talk) 16:58, 2 June 2011 (UTC)

WRT this medicinal mushrooms review articles do exist and they are much less rosy than the primary research. If reviews did not exist and the subject got a lot of lay press than yes but creating compilations of all the positive research on a subject is not NPOV. Doc James (talk · contribs · email) 21:59, 2 June 2011 (UTC)

Yes, I agree: Once decent reviews exist (or once decent primary clinical studies exist, for stuff previously sourced to petri dishes and animal studies), then the older, weaker sources should be replaced by the stronger, more relevant sources. WhatamIdoing (talk) 01:16, 3 June 2011 (UTC)

Again

Could we please have a clear recommendation in WP:MEDRS what to do with primary sources prior to the existence of reviews? I have just witnessed another edit removing four primary sources only to be replaced with a {{fact}} tag. My opinion on this is: by all means replace the primary sources with a review, but replacing with a {{fact}} tag is distinctly unhelpful. pgr94 (talk) 14:07, 28 July 2011 (UTC)

It looks like the article in question is Hypothyroidism. For the most part, it looks like the other editor was just screwing up the citation formatting (e.g., deleting everything except the doi), and that a bot came by and fixed it again. However, here, he deletes about 15 citations to a statement that says "This has been investigated in more than a dozen studies, most of which conclude that a combination therapy of T4 and T3 does not appear to be superior to monotherapy."

What the editor should have done is leave the citations (which could have been put in a single ref tag to avoid the appearance of citation overkill) alone and tag the sentence with {{better source}}. However, he probably didn't know that this template exists.

Are you sure that no secondary source summarizes these studies? WhatamIdoing (talk) 15:07, 28 July 2011 (UTC)

i know about this template, but original research / synthesis from 12 sources is just so completely unhelpful that i think it's better to just delete and start over. —Chris Capoccia ^T⁄_C 16:05, 28 July 2011 (UTC)

This is not just about hypothyroidism; I've seen it happen a few times. I'm uncomfortable with primary sources being deleted and not being replaced by something better. The main reasons are: 1) it leaves the claim with no support at all; and 2) it makes it harder to find a secondary source. (Reviews can be found by searching for articles that cite the primary articles). pgr94 (talk) 16:27, 28 July 2011 (UTC)

Chris, I'm not sure that actually qualifies as a SYNTH violation. It's not a SYNTH violation of any educated person could look at the sources and find the claimed material. It's also not a SYNTH violation to provide a plain description of primary sources.

Do we agree that any educated person can see from this list that there were more than a dozen studies on the subject? And that any educated person could look at the studies and see that "most" (=eight or more of the fifteen cited sources) conclude that combination therapy does not appear to be superior? WhatamIdoing (talk) 16:29, 28 July 2011 (UTC)

now that i've carefully looked at the sources, it most definitely IS synthesis. the actual conclusion should be "no consensus" and not for or against. i'm putting the mess back (slightly reformatted to organize by conclusion) in the hopes that this will help identify quality seconary sources. —Chris Capoccia ^T⁄_C 18:28, 28 July 2011 (UTC)

If a majority of the sources cited do not come to the conclusion that the text claims that they do, then it is not SYNTH: it is merely wrong. WhatamIdoing (talk) 18:41, 28 July 2011 (UTC)

I'm sorry, but why are we using primary studies when we should be using reviews like this and this? Yobol (talk) 20:29, 28 July 2011 (UTC)

thanks for finding those. i've switched that section. —Chris Capoccia ^T⁄_C 20:53, 28 July 2011 (UTC)

Clarification

A few editors have been removing organizational policy information from Circumcision based on MEDRS. They say if its over 3-5 years old it should be removed while I have contended that MEDRS is referring to science and has nothing to do with organizational policies. Garycompugeek (talk) 13:58, 13 July 2011 (UTC)

Well, that's a... partial description of the dispute.

Considering this diff, it appears that you're trying to use a 1999 (twelve years old) paper to make a statement including the words "Virtually all current policy statements from specialty societies and medical organizations..."

Now I don't know (or care) much about this subject, but it does seem to me that the recommendations on neonatal circumcision have changed in the last twelve years, specifically in areas with very high HIV rates. This statement, while doubtless accurate about the state of recommendations in 1999, has very likely been overtaken by events and is very likely no longer accurate. Presenting it as if nothing could have changed in the meantime is inappropriate. WhatamIdoing (talk) 15:37, 13 July 2011 (UTC)

Aside from WHO no one else has changed or updated their circumcision policy that I am aware of so the statement still seems pretty accurate to me. Garycompugeek (talk) 17:24, 13 July 2011 (UTC)

Even if zero of them changed their policies, a twelve-year-old document is not an ideal source of information for any type of claim about current policy. WhatamIdoing (talk) 17:29, 13 July 2011 (UTC)

I disagree. Current policies are the practicing polices of an organization regardless of age until a new policy is set into place. Garycompugeek (talk) 19:58, 13 July 2011 (UTC)

All the WHO gets is an aside? They are the most recognized health institution in the world. A bunch of other groups have changed their position aswell. Which one can find in a number of the review articles.Doc James (talk · contribs · email) 20:08, 13 July 2011 (UTC)

Gary, the AMA quotation isn't primarily about their own position. It's about the positions held by many other organizations. The AMA statement about the other groups becomes wrong the day that some other group changes its position, not on the day that the AMA changes the AMA's position.

To give an example: I've got a 15-year-old source that says Elizabeth Taylor is married. That doesn't mean that she's still married: During the last 15 years, she got divorced and died. Taylor doesn't remain alive and married because an old source said she was alive and married back then. The positions of the various groups don't remain what they were in 1999 merely because an old source tells us what those positions were back then. WhatamIdoing (talk) 22:25, 13 July 2011 (UTC)

Thank you for your feedback WhatamIdoing. I understand your point. I simply havn't seen any dramatic shift of circumcision policy from major medical organizations, even WHO was only recommending it in high risk HIV areas. To me the word "virtually" stills seems to hold true. My original question has still went unanswered. Does MEDRS 3-5 year rule apply to science and policy or just science as I have contended? Garycompugeek (talk) 13:17, 14 July 2011 (UTC)

I agree with Gary. I live in Britain and the largest medical organization NHS has maintained their original position: Namely that the "the majority of healthcare professionals now agree that the risks associated with routine circumcision, such as infection and excessive bleeding, far outweigh any potential benefits." Most other European groups have said the same. Pass a Method talk 09:12, 15 July 2011 (UTC)

Even if that were a policy statement (which it appears not to be), it would only be a single policy, so one cannot reasonably infer anything about other policies from it. Jakew (talk) 10:34, 15 July 2011 (UTC)

Jake, i dont relly understand what you meant by "other policies", but here is the statement on their website.[10] Pass a Method talk 12:43, 15 July 2011 (UTC)

Yes, as I thought it's just a web page, not a formal policy approved by membership or a policy committee. But since you didn't understand my more important point, let me try again. Suppose we have a group of, say, 30 students, and we wish to determine whether showing them a provocative and challenging film changes their viewpoint about freedom of speech. Now suppose that we interview one student (let's call her Sally) afterwards and find that her view is unchanged. Can we legitimately conclude that none of the students' views have changed? It seems obvious to me that we can't: Sally might well be an exception (conceivably she might even be the only student whose views haven't changed). A sample of one, random or otherwise, is too small to reliably extrapolate to most, let alone the entire group. So, returning to the subject, even if this were the NHS's formal policy, we couldn't draw any conclusions about "virtually all" medical associations from it. Jakew (talk) 15:22, 15 July 2011 (UTC)

Just a webpage? You're wrong. If its on the official website of a government health service, then its obviously a policy statement. Seems that you're just playing with words here to me. This policy is backed up by other government funded media such as BBC [11]. In other words, the British government policy consensus is against routine circumcision. Both the BBC and NHS are publicly funded government organizations hance they are impartial on this issue. Plus these webpgaes are regularly updated. Pass a Method talk 20:02, 15 July 2011 (UTC)

I'm afraid you're mistaken. Not everything published by an organisation (on its website or elsewhere) constitutes formal, authoritative policy of that organisation (there is also a distinction between types of information published by organisations, as explained in WP:MEDRS#Medical and scientific organizations). Jakew (talk) 20:43, 15 July 2011 (UTC)

Gary, you say 'To me the word "virtually" stills seems to hold true'. That may be so, but if Garycompugeek is a WP:RS then we might as well cite him directly. On the other hand, if Garycompugeek isn't a reliable source then it really doesn't make any difference what he thinks about the current state of policy statements. So we're back to the original question: how much weight should be given to a twelve-year old source regarding current policies? Jakew (talk) 10:34, 15 July 2011 (UTC)

Any possibly outdated source should be replaced by a current one. It doesn't matter what the subject is: outdated sources should be shunned for Liz Taylor, and they should be shunned for Microsoft, and they should be shunned for circumcision. This isn't a rare disease in a neglected field; it should be very easy indeed to find a more recent statement (which might well say almost the same thing, but which would at least give the reader some confidence that a reliable source very recently said this, rather than just one of us saying that probably nothing important has changed in the last dozen years).

On the underlying point: Properly speaking, one does not have a policy on what other groups' policies are. It's silly. One reports, in background sections, what one believes to be the most important facts relevant to a situation, which in most medical guidelines and policy recommendations includes facts about what other groups recommend.

"Most groups recommend ____" is not a policy statement: It's a fact, and as such the statement can be either "true" or "false". (We believe it very safe to assume this reported fact was true at the time; it might still be true now, but we don't know what might have changed in the last dozen years.) "We recommend ____" is the actual policy statement. The facts can change at any time and without any action on the part of the policy maker, but the policy itself cannot without action by the policy maker.

Since we're aware of one major body changing its position recently, it seems like the best possible solution would be to find a source that both (1) summarizes the overall state of official opinion and (2) was clearly written after the WHO's policy change was announced. WhatamIdoing (talk) 16:23, 15 July 2011 (UTC)

It is simple. Buck up, do some research and find current sources to support the content. I have no special opinion on this topic. But it still needs to follow Wikipedia policy like any other topic area. If people find current high quality research I will support its inclusion no matter what the conclusions are.Doc James (talk · contribs · email) 16:49, 15 July 2011 (UTC)

This has been recently done by Lizard Wizard already with a 2010 source. No hard feelings Doc and I appreciate your support of the new source. I simply wanted to keep an accurate summation of prevalent medical opinion in the intro. Garycompugeek (talk) 19:25, 15 July 2011 (UTC)

Continuing education

I'm currently assuming that journal papers published under the rubric "Continuing Education" can, like those under "Review" be treated as MEDRS, but I don't see anything in the guideline that makes that explicit. Am I right, and should it be spelled out? LeadSongDog come howl! 15:58, 20 July 2011 (UTC)

Can you give an example? I can think of CME stuff that might technically (barely) be reliable, but that I wouldn't really want to use. Most of what I'm thinking of isn't really a "journal paper", though. WhatamIdoing (talk) 20:56, 26 July 2011 (UTC)

The case in point was: Donatini B. "Le Hericium erinaceus: des propriétés essentiellement dépendantes du neuronal growth factor" [Hericium erinaceus: properties mostly related to the secretion of neuronal growth factor] Phytothérapie ISSN 1624-8597 (2011) 9(1):48-52, doi:10.1007/s10298-010-0601-6 (Formation Continue [Continuing Education])

LeadSongDog come howl! 17:19, 28 July 2011 (UTC)

Having looked over a machine translation, I have pretty mixed feelings. It's secondary and independent, which is all good. But I wonder how credible the journal is (it's not indexed at PubMed), and I can't find any reference to the author's institution outside of several very similar publications by the same person (might be a PR or information agency?). So if this is typical, then maybe it's better than a string of primary sources, but I'm not sure that it's really equivalent to a proper literature review article. WhatamIdoing (talk) 18:25, 28 July 2011 (UTC)

The journal, like most of Springer's is widely indexed [12], but that wasn't really the thrust of the question. Even if that journal lacks substance (I tend to think it's roughly as solid as CAM journals get) we may still need to address the gap in this guideline. If it had been a cardiology journal rather than phytotherapy, the same question would still stand.LeadSongDog come howl! 22:13, 28 July 2011 (UTC)

Identifying a systematic review

How does one identify a systematic review? The guideline says this: " A systematic review uses a reproducible methodology...." My assumption based on that statement is that if a research review uses a reproducible methodology, then it's a systematic review. What I have found is that such reviews typically begin with a methods section that outlines the specific databases that were searched and the keywords that were used. And then discusses the results of the search, explaining which studies were excluded from consideration and why. If a research review does these things is it a systematic review? An editor is arguing that in order to be a systematic review, the review itself must explicitly say it's a systematic review. If that's true, then I suggest that the guideline be changed so that editors know that systematic reviews always self-identify as such. Thanks. TimidGuy (talk) 10:44, 28 August 2011 (UTC)

There are a number of components including "critical appraisal". It is more than just looking through a database for key words. [13][14] Doc James (talk · contribs · email) 11:06, 28 August 2011 (UTC)

Thanks, James, for two interesting links. Hordaland (talk) 18:05, 28 August 2011 (UTC)

I guess I'm still unclear. Must a research review explicitly state that it's a systematic review? If not, should we spell out the criteria in this guideline so that editors can identify which reviews are systematic reviews? This guideline values systematic reviews above narrative reviews, so it would be good that it be clear on how to identify them. TimidGuy (talk) 09:50, 29 August 2011 (UTC)

A systematic review will state that it is a systematic review else it is not. Doc James (talk · contribs · email) 09:54, 29 August 2011 (UTC)

Thanks. Then we should state that in the guideline. Is there consensus for this? Maybe something like: "Although some narrative reviews use aspects of the methodology of systematic reviews, such as a methods section that describes the literature search, true systematic reviews explicitly self-identify as such." TimidGuy (talk) 10:16, 29 August 2011 (UTC)

I'm not sure that's helpful. This might be the convention, but that wouldn't prevent someone in a fringe journal from abusing the label, and it doesn't guarantee that every single systematic review will actually contain those words.

I don't think that we really intend to value systematic reviews "above" narrative reviews. They have different purposes. If your question is along the lines of, "Does administration of corticosteroids to women giving birth prematurely save their babies lives?", then the systematic review is your best option. But if your question is less specific than that—what are the general causes of premature birth, what are the commonly accepted risk factors, how frequent is this, what are the typical complications, what kind of supportive treatments might be needed, what are the risks to the mother—then the narrative review (or good textbook) is your best choice. WhatamIdoing (talk) 18:36, 2 September 2011 (UTC)

Thanks. And my impression, having looked at a couple hundred research reviews, is that not every systematic review self-identifies as such. Plus, an editor involved in the dispute with Doc James said that this was confirmed by a medical reference librarian. TimidGuy (talk) 09:42, 3 September 2011 (UTC)

Since we are bringing in the significance of outside peoples opinion people deserve some background to your question. It is likely that [...][ the confirmation is from the medical reference librarian at the MUM. For some background User:TimidGuy [...] [...]. He has also spend much time trying to get and keep probably the best piece of research from the article [15] a systematic review and meta analysis on the topic at hand by the AHRQ. Here are some of his edits in which he removed said piece of research [16] and [17] He has attempted many time to move this research to a less prominent place within the article, to reduce the emphasis put on it [18] and now since that has not succeeded tagged all TM article with NPOV tags.[19] So simply put Timidguy wishes to use literature reviews which give positive results of TM in passing to refute the conclusions of the Cochrane reviews and AHRQ meta analysis regarding health effects of TM. Here is the study that is being claimed to be a systematic review. Doc James (talk · contribs · email) 10:22, 3 September 2011 (UTC)

Ethnicity and Disease,Paradies, Yin (2006). "A Review of Psychosocial Stress and Chronic Disease for 4th World Indigenous Peoples and African Americans" (PDF). Ethnicity and Disease. 16: 295, 302, 305. Retrieved March 31, 2011. {{cite journal}}: Unknown parameter |month= ignored (help) (independent systematic review) Quotation: “In general, TM was more effective that PMR (progressive muscle relaxation) in an eight-year follow-up of 530 African American participants which found a 63% reduction in all-cause mortality and an 82% reduction in heart disease mortality in the intervention group compared to the control groups. . . . Six randomized conrolled studies . . . used TM as the intervention, with PMR also used as a secondary intervention in several trials . . . Compared to the control group, those who undertook the TM intervention had decreased carotid intima-media thickness, systolic blood pressure, and diastolic blood pressure, heart rate and cardiac output, anxiety, depression, neuroticism, and sleep dysfunction, as well as increased energy, general mental health, and health locus of control.

Here is a Cochrane review that disagrees with the anxiety bit and the AHRQ disagrees with the rest. Krisanaprakornkit, T (2006 Jan 25). "Meditation therapy for anxiety disorders". Cochrane database of systematic reviews (Online) (1): CD004998. PMID 16437509. {{cite journal}}: Check date values in: |date= (help); Unknown parameter |coauthors= ignored (|author= suggested) (help) Doc James (talk · contribs · email) 10:46, 3 September 2011 (UTC)

The African review is not a systematic review. It never claims to be one, and it says "[Given several problems with the studies] it was not possible to perform a meta-analysis with the studies identified in this review. For similar reasons, it was not thought that an indepth examination of the methodological strengths and weaknesses of the studies in this review would be appropriate." --Enric Naval (talk) 11:14, 3 September 2011 (UTC)

Thanks, Enric. It's good to get clarification regarding what constitutes a systematic review. Which of those statements indicates that it's not a systematic review? Is a systematic review always a meta-analysis? Must a systematic review have an in-depth examination of the methodological strengths and weaknesses? Also, do you feel that it's accurate that a systematic review will always identify itself as such? TimidGuy (talk) 11:22, 3 September 2011 (UTC)

I'm fairly sure that systematic reviews are supposed to evaluate the quality of reliability of the studies, as opposed to simply summarizing the results and dumping together good-quality studies with bad-quality ones. Then they are supposed to give them appropriate weight when reaching conclusions. At what point that becomes a meta-analysis, you will have to ask someone more familiar with the topic. --Enric Naval (talk) 11:42, 3 September 2011 (UTC)

A meta-analysis typically pools the data from multiple studies and does a separate analysis to see if there's a statistically signifiant effect. The AHRQ review that Doc cites included both strong and weak studies in the meta-analysis. Some meta-analyses do subgroup analyses of the strong studies and the weak studies and then compares the results, but AHRQ didn't do this. Can we conclude that a systematic review isn't necessarily a meta-analysis? TimidGuy (talk) 11:53, 3 September 2011 (UTC)

In a broad sense, a meta-analysis is a type of systematic review, because it is usually necessary to systematically identify studies, extract data from them, and synthesise it to form a summary estimate. That is, after all, a systematic process for reviewing the papers. However, it's not what people usually mean by "systematic review". The results of meta-analysis are typically published in the same paper as a (qualitative) systematic review of the source materials, but that isn't always the case. Incidentally, it's also possible to find meta-analyses that are non-systematic (such as those that involve an arbitrarily selected set of source papers). Jakew (talk) 12:17, 3 September 2011 (UTC)

A systematic review is not necessarily a meta-analysis. A systematic review can involve no number crunching (many don't, especially if zero or only one paper is found that meets the conditions, or if the underlying data is unavailable or unsuitable for one reason or another), and, as Jakew says, a meta-analysis might use only data chosen non-systematically (e.g., a meta-analysis of publications I recall reading off the top of my head).

What makes something a systematic review is that it follows (and describes) a system for including and excluding studies. Whether that system is a good one is still something that the reader will have to decide (after all, I could have a "system" of "Search on PubMed for studies using both this keyword and my own name", which would hardly result in the lack of bias usually intended for such a study), but the defining characteristic of a systematic review is a system.

As for the specific study under dispute, if you aren't satisfied with the word of one librarian, then go ask another. I usually inflict librarian questions on User:DGG. You might also try User:Trystan, who also appears to be a librarian. WhatamIdoing (talk) 23:06, 3 September 2011 (UTC)

Thanks much for the feedback. Is there a change that we should make to the wording of this guideline to help clarify what one should look for in order to identify a systematic review? TimidGuy (talk) 10:08, 4 September 2011 (UTC)

If someone has ready access to a copy, I believe this distinction should be discussed in:

Style Manual Committee (June 2006). Scientific Style And Format: The CSE Manual for Authors, Editors, And Publishers (7th ed.). Council of Science Editors. p. 3. ISBN 0-9779665-0-X.

(not seen) LeadSongDog come howl! 15:04, 12 September 2011 (UTC)

Opinions sought

A discussion with Doc James has raised two points that may be worthy of discussion here

1. Should case reports of unusual presentations of a rare disease with a known cause be citable on Wikipedia?

2. Given that a number of featured articles include primary source material rather that secondary sources should these articles be downgraded until this material is expunged?

Many thanks for your time. DrMicro (talk) 12:20, 2 September 2011 (UTC)

For point 1, the relevant policy is WP:NPOV, particularly the WP:WEIGHT section. If "case reports of unusual presentations of a rare disease with a known cause" are worthy of mention on Wikipedia, then they will also be mentioned in the appropriate secondary literature on the disease. If they are mentioned, then we should consider the weight given to such reports (for example, a simple summary sentence mentioning the several presentations, rather than many sentences or a section). If they are not mentioned by other authors, then it is fairly likely that we should also not mention them.

For point 2, issues you find with specific articles should be raised on those article's talk pages and optionally also at WP:MED. Should an article be felt to be significantly below the standard required of an FA, and the issues raised on the talk page are not being addressed, then it can be taken to WP:FAR (taking care to note the procedure there, and avoiding WP:POINT). Colin°^Talk 13:24, 2 September 2011 (UTC)

Concerning point 2 perhaps I was not sufficiently clear. Doc James is of the opinion that the inclusion of primary source material should prohibit an article from ever becoming a featured article until such time as this material is removed regardless of its other merits. This is to a general policy applicable to all articles. For this reason it does not seem necessary to discuss this on the individual pages.DrMicro (talk) 15:49, 2 September 2011 (UTC)

You misrepresent Doc James' views. He said "Primary research should only occasionally be used", and he limited that statement specifically to medical claims. "Occasionally" is a far cry from "never", and even a plainly medicine-related article will contain non-medical claims. For example, Schizophrenia, which was beautifully cleaned up a while ago, has a handful primary sources at the end; they are about cultural and economic issues. WhatamIdoing (talk) 18:57, 2 September 2011 (UTC)

With all due respect to WhatamIdoing I am not sure that I did misrepresent Doc James. Of course only he can say for definite what he meant but I did refer these questions to him and he seemed contented with what I had written. I would assume - possibly incorrectly - that he did not then consider that he had been misrepresented. I have had quite a long discussion with him over this matter and that appears to be his viewpoint. I did draw to his attention that all the featured articles I sampled - presumably a fairly random sample - seem to have citations to primary sources. Despite this it was I believe his opinion that all this material should be replaced with secondary source citaions if these articles are to reach or maintain featured article status.DrMicro (talk) 19:09, 2 September 2011 (UTC)

Concerning point 1: The WP:WEIGHT and WP:NPOV refer to subjective viewpoints rather than to the inclusion (or non inclusion) of facts. For this reason these pages may or may not be relevant.DrMicro (talk) 15:49, 2 September 2011 (UTC)

Perhaps the most relevant material is taken from this policy statement:

"A primary source may only be used on Wikipedia to make straightforward, descriptive statements that any educated person, with access to the source but without specialist knowledge, will be able to verify are supported by the source."

A case study describing an unusual presentation of a disease would seem to fit this description. DrMicro (talk) 15:57, 2 September 2011 (UTC)

Some featured articles were promoted a long time ago, when standards we less strictly enforced. Also, unless a featured article is carefully tended, there is a natural decay where inferior material creeps in. Sadly, this can grow to the point where the article needs a lot of work to meet FA standards, and the article gets demoted. Doc James is right that any violation of policy could prevent an article becoming featured. We don't say, "This is so well written, that it doesn't matter if a couple of paragraphs are unsourced", for example.

Most of NPOV/WEIGHT concerns viewpoints, but it also concerns the whole balance of an article content:

An article should not give undue weight to any aspects of the subject but should strive to treat each aspect with a weight appropriate to its significance to the subject. For example, discussion of isolated events, criticisms, or news reports about a subject may be verifiable and neutral, but still be disproportionate to their overall significance to the article topic. This is a concern especially in relation to recent events that may be in the news. Note that undue weight can be given in several ways, including, but not limited to, depth of detail, quantity of text, prominence of placement, and juxtaposition of statements.

There are many facts and opinions that WP could note, but deciding which to include and to what degree is done by comparison to the existing published literature. WP:PSTS says "Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources." There are cases where primary sources can be used but these are rare. Our Wikipedia:Core content policies should "not be interpreted in isolation from one another". The sentence you quote restricts the use of primary sources, it doesn't encourage them. Colin°^Talk 16:55, 2 September 2011 (UTC)

The question arose from a series of unusual case presentations including one of cocoon syndrome where the correct diagnosis was life saving. These are all recent (< 2 years old) and I doubt anyone has checked all the reviews since then to see if they were cited.DrMicro (talk) 19:03, 2 September 2011 (UTC)

If those unusual presentations are so atypical that every single secondary source has declined to mention these case reports, even in passing (and we've waited long enough to give them a fair chance to do so), then they are probably too unimportant to mention. This is an encyclopedia, not a compilation of every single verifiable detail.
Not all primary sources are as flimsy as single-patient case reports. Some uses of some primary sources are appropriate, e.g., when the primary source provides more detail about something that a secondary source mentions in passing. Furthermore, when inappropriate uses of any source are discovered, the appropriate response is to fix it already, not to "downgrade" the article. WhatamIdoing (talk) 18:48, 2 September 2011 (UTC)

I for one am inclined to agree with WhatamIdoing over the use of primary source material. The question is what status should these articles have until the primary source material is sourced. Should they be downgraded pending such a replacement? Should WP adopt a policy of all new articles aiming for featured status have all the primary source data deleted or replaced before such an upgrade be made while leaving existing featured articles with their current status? Is WP too rigid over its choice of citable material?

WhatamIdoing has raised a second point here. Recent case reports which may be of interest may simply be too recent to be included in reviews. Usually it takes several months for a review to be published (depends on the journal) and that assumes that the author(s) have read all the recent material. Additionally how many editors read ALL the available reviews on a topic? Or even a sufficient number to say with confidence that such material has not been included in a review.DrMicro (talk) 19:03, 2 September 2011 (UTC)

Our policies and guidelines apply to all articles, whether applying for featured status or good-article status or just a stub. We don't downgrade articles pending some fix, we try to implement the fix (either ourselves or by using the talk page or help from a project). Demoting articles is a last-resort step. Wrt recent stuff, see the policy text I quoted and the essay WP:RECENT. We are in no rush. We are not the news. Guidelines are guidelines, of course, so if there are specific problem articles or sources, then raise the issue at WP:MED and see if editors there can help. Colin°^Talk 19:52, 2 September 2011 (UTC)

What you call "downgrades" take weeks, and sometimes months, to achieve. It's not merely a matter of one person saying "Oops—I found a problem, so I'll de-list it until it's fixed!" If you tried to do that, you'd quickly get a reputation for being a lazy, uncollegial, and disruptive editor. I'm sure that's not what you want, and I'd rather have you doing something useful instead of dealing with the inevitable backlash.

WP:Good article status is not required to comply with this particular guideline (GA is very limited in its requirements), so this guideline cannot result in de-listing. WP:Featured articles are. Both are required to comply with WP:NPOV—but there simply is no rule, here or anywhere, that says all information supported by primary sources must be deleted, so if you showed up at FAR with a demand that any article be de-listed merely for containing a primary source, you'd get laughed at. On the other hand, if you show up at FAR with a complaint that it seriously over-relies on primary sources, it will be taken seriously—but seriously enough to fix it eventually, as was done with Schizophrenia, not to de-list it immediately.

You might think this rule "too rigid" since you've encountered opposition to citing a long string of single-patient case reports from a couple of years ago, but I doubt that very many people would agree with you. Wikipedia is not a collection of case reports. That information might be welcome at WikiBooks, but it's not part of an encyclopedia article. We're not trying to save people's lives by telling them that some unusual manifestation was once associated with a given disease. We're trying to give the average reader a general overview of the subject. One method we have for distinguishing trivia from more important information is whether the information is present in a secondary source. (It's not the only method; e.g., more important information is normally present in multiple sources, and it is often discussed in detail, while trivia is often mentioned only in one source, or in passing.) This is a pretty basic application of the WP:DUE policy. I'm sorry that it's not letting you include whatever detail happens to interest you, but that's a natural consequence of the goal, which is an encyclopedia article, not a comprehensive catalog. WhatamIdoing (talk) 20:00, 2 September 2011 (UTC)

To clarify this point it was not I that suggests that featured articles should not include primary sources. Rather I suggested the opposite. However it did occur to me that the suggestion did have some intrinsic merit being clear and easy to implement so I proposed to seek second opinions here. It would appear that primary source material is acceptable in featured articles and by extension in other articles. This was my understanding and I am glad to see that other editors here agree with me.

Concerning 'a long string of case reports': One report was a number of cases of gastroenteritis in children. A second was a rare syndrome of generally unknown cause where the identification of the organism was literally life saving. The report of agstroenteritis in childen is consistent with the idea that this organism is a common environmental organism that most children and adults gain immunity to early on. The total number of case reports was four in all which seems a little difficult to describe as a long series. But your mileage may differ.DrMicro (talk) 22:36, 2 September 2011 (UTC)

Perhaps I should also mention that these cases were listed under "unusual presentaions". This was done to try to ensure that a balenced view point was maintained.DrMicro (talk) 22:45, 2 September 2011 (UTC)

"[i]f you showed up at FAR with a demand that any article be de-listed merely for containing a primary source, you'd get laughed at." Thank you WhatamIdoing. That was and is my opinion also. Which is again not to say that Doc James suggestion is entirely without merit and may be worthy of consideration in a future interation of the guidance on such articles.DrMicro (talk) 22:43, 2 September 2011 (UTC)

DrMicro, "the opposite" of "featured articles should not include primary sources" is that they should include primary sources. This is quite wrong. Let me repeat what WP:PSTS says: "Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources." The use of primary sources, primary research papers or case studies, is exceptional. Saying "It would appear that primary source material is acceptable in featured articles and by extension in other articles" is a naive conclusion along the lines of saying "It would appear that in the UK it is acceptable to drive on the right-hand-side of the road". There are circumstances where driving on the other side of the road is the correct action but it isn't normal and one can usually get from A to B without doing so.

As for your list of unusual presentations, sourced only to case studies, it is merely your opinion that the inclusion of these is balanced. But per WP:WEIGHT what counts is the balance indicated by reliable sources, not the opinions of editors. If you showed up at FAC (FAC, not FAR) with an article containing a list of medical facts sourced only to primary research papers or to case studies, then it would fail. Continuing to argue the case that these unusual presentations should be included is a pointless task and wikilawering to try to get them included doesn't impress. You need to show that these presentations are worthy of inclusion because other reliable sources, when discussing the article topic, also include them. That's policy. It applies to all articles on WP. Colin°^Talk 08:59, 3 September 2011 (UTC)

It seems desirable to know precisely what was intended by the use of the term 'impress' here as it is not clear. While it is probably redundant to remind anyone of the the policy of civility on this site, it may still be be necessary. Reasonable people may hold differing opinions on subjects and still remain civil to one another. It is also probably redundant to point out that this is a talk page where these matters are supposed to be discussed. If there is some other forum where this matter should instead be discussed it would be useful to provide a link to the relevant talk page.

It is my current opinion that clarification on the two matters I raised here are now clear: firstly case reports should not be included unless referred to within a secondary publication. Secondly primary source material may be included with articles including featured ones and that inclusion of such material is on a case by case basis: in short the proposed policy of banning by Doc James all primary source material as part of the process towards a featured article is not currently regarded as desirable.

If this is an incorrect summary of the current position I would be grateful if another editor(s) would clarify this position.DrMicro (talk) 13:02, 3 September 2011 (UTC)

The use of primary sources should occur only rarely and only in exceptional circumstances. Most editors should be using secondary sources 99% of the time. If there is controversy surrounding primary sources they should generally not be used and secondary sources should be found. Doc James (talk · contribs · email) 22:14, 3 September 2011 (UTC)

I don't think we need to be completely primary-source-phobic. Properly used, primary sources can add a lot to an article. Unfortunately, we've had a lot of problems with editors selectively choosing primary sources to advance an agenda or to try to "rebut" expert scientific opinion. It's very easy to do - too easy, really - and thus the cautions about using primary sources. But the problem only really arises when primary sources are being misused. MastCell ^Talk 23:05, 3 September 2011 (UTC)

And, as proven by Doc James' comment here, DrMicro is wrong when he says that Doc James has proposed banning all primary source material from any article.

I want to repeat an earlier point: There are many kinds of primary sources, and they are not identical in their utility. Single-patient case reports are among the very least useful primary sources. Large, multi-center, randomized, placebo-controlled clinical trials like the Heart Protection Study also result in primary sources (e.g. PMID 12114036), and these can occasionally be valuable (especially in the short-term, before the studies have been replicated). "Shouldn't use those particularly weak primary sources" does not equate to "Must never use any primary sources at all". WhatamIdoing (talk) 23:17, 3 September 2011 (UTC)

Here's an idea. Wikipedia operates by consensus, and that consensus can include proscribing certain types of sources and stipulating definitions. Clearly the consensus among those who created this guideline is that case reports don't meet a minimum threshold. This is parallel to the community's consensus that, for example, blog comments don't meet a minimum threshold. Perhaps we could simply stipulate that a case study doesn't necessarily even have the status of a primary source, just as we have stipulated that regarding blog comments. TimidGuy (talk) 10:18, 4 September 2011 (UTC)

The problems with blogs are different to that with case studies. Blogs have a WP:V problem in that we don't regard them as reliable for much other than perhaps the opinion of the blogger, should they be notable. Case studies have a WP:WEIGHT problem but are reliable sources of minor, specific and non-generalisable information. The confusion might be that we use the word "reliable" when judging evidence that a finding is likely to be significant and generally applicable, and case studies are a poor source of evidence. This guideline already notes that. For example, when describing the history of a disease, one might use details sourced to the historical case reports to flesh out the story. Using primary sources to supplement the core, which is solidly founded on secondary sources, is quite reasonable, though even that may not be necessary for a well published topic. Colin°^Talk 13:56, 4 September 2011 (UTC)

I like your example of how a case report might be used. I hadn't thought of that. And I had wondered if it was okay to go back to the study itself for a little more detail, such as what comparator was being used. Glad to hear that you think that's reasonable. TimidGuy (talk) 19:05, 4 September 2011 (UTC)

Finding Secondary Sources

Could someone tell me how someone goes about finding secondary sources? Arydberg (talk) 09:18, 5 October 2011 (UTC)

Read Wikipedia:Identifying reliable sources (medicine). It contains the information you require. Colin°^Talk 12:31, 5 October 2011 (UTC)

Proposal stemming from recent Signpost editorial

Per this discussion, I would like to propose adding the following sentence to this guideline: "Unsourced or poorly sourced biomedical content should be tagged or removed." Thoughts? Nikkimaria (talk) 14:16, 3 November 2011 (UTC)

I'd rather see this, or somthing similar, added to WP:BURDEN (part of WP:V) first. Also, the BLP policy only refers to material "if it might damage the reputation of living people" rather than anything one might say about a living person. Only a minority of text in a "biomedical" article, if wrong or misleading, could actually lead to harm. There are more ways of causing harm in a biomed article than just saying nasty things about a person. So identifying the areas where a zero-tolerance policy can apply could be tricky.

BTW, one reason we don't give dosage information on WP (yet get criticised for this when compared to other reference works) is because it would be nearly impossible for us to specifiy this accurately without reproducing the entire BNF/etc and without an army of volunteers to police all the tens of thousands of drug articles to make sure the 150mg wasn't 15mg. Colin°^Talk 17:06, 3 November 2011 (UTC)

So much "biomedical" information is inconsequential that it's not appropriate. We once had someone tag a sentence that reported the typical number of fingers on the human hand because it was unsourced. That's "biomedical" information. Do you really think that we need an inline citation for a fact that is far more obvious than the fact that the sky is (normally) WP:BLUE?

People need to use their best judgment. Tag-bombing every single unsourced claim in an article is not helpful. In fact, by hiding the serious problems in a sea of indiscriminate tagging, it's actually unhelpful. WhatamIdoing (talk) 17:36, 3 November 2011 (UTC)

Okay, then let's add potentially harmful to the proposal. Nikkimaria (talk) 17:51, 3 November 2011 (UTC)

Whatever we come up with here, the discussion should eventually go to a more prominent place (Village Pump, Centralized discussion template, etc), and be added to a more prominent page (anyone notice that WP:V doesn't even link to MEDRS, which is why I constantly deal with poorly sourced additions on my watchlist from university classes?) But, there is a lot of highly dubious info in biomed articles, so we have to understand what we mean by "potentially harmful"-- I personally think just about any biomed misinformation is potentially harmful, and I've worked on scores of articles that would be better served by being gutted than by trying to fix them. SandyGeorgia (Talk) 18:13, 3 November 2011 (UTC)

Actually, you added a link to MEDRS in September 2010, and it's been there ever since. Nikki added a second link recently. WhatamIdoing (talk) 21:59, 10 November 2011 (UTC)

One of the big problems we have is people attempting using primary research in an attempt to refute reviews even when reviews are present. Than repeatedly reverting when reviews are added claiming that consensus is needed. This is a current issue at Vitamin D.Doc James (talk · contribs · email) 00:13, 5 November 2011 (UTC)

Even perfectly accurate biomedical information is "potentially harmful". This complaint is a simple illustration of the problem. WhatamIdoing (talk) 21:50, 10 November 2011 (UTC)

Secondary sources verses a paper by the Institute of Medicine

A argument is being put forth here [20] that a report from the Institute of Medicine should be used to the exclusion of a 2011 review article in the Lancet. That it as a "tertiary source" should trump secondary review articles. Reverting the content between two versions continues and added input would be appreciated.Doc James (talk · contribs · email) 13:53, 6 November 2011 (UTC)

MEDRS states:

Ideal sources for such content includes general or systematic reviews published in reputable medical journals, academic and professional books written by experts in the relevant field and from a respected publisher, and medical guidelines or position statements from nationally or internationally recognised expert bodies.

Both sources are "ideal". Generally, a massive report produced by a respected national body and including the views of many experts from many institutions could be assumed to be of higher standing than a review written by a few from one institution and published in a medical journal. Having said that, a review in The Lancet should not be discarded lightly. The secondary/tertiary distinction is not at all relevant to this discussion. Both sources have experts trying their best to weight the evidence before them and come to a conclusion. That the IoM report has proved extremely controversial doesn't surprise me given the size of the supplement industry and its negative impact on them, but this isn't the place to discuss the details of Vitamin D or supplementation or individual sources. The only thing we can say is that MEDRS thinks these sources are great. You can't expect a WP guideline to help resolve the "which one is right" battle, especially if the real world hasn't solved that battle either. Colin°^Talk 15:30, 6 November 2011 (UTC)

Can we at least agree that both can be used? That is my interpretation of this guideline. Especially since the IoM did not look at research after 2009 for MS and more recent reviews have.Doc James (talk · contribs · email) 15:38, 6 November 2011 (UTC)

I agree with Doc. James. Censoring any highly-credible source dis-serves the credibility of the encyclopedia. Put all the highly-credible sources in. — Preceding unsigned comment added by Ocdnctx (talk • contribs) 21:39, 19 December 2011 (UTC)

I don't think it's primarily the "supplement industry" fueling the controversy. If you read about the controversy, you do get a clear impression that there is a real scientific dispute here. The mere existence of a scientific paper with the title "IoM's utter failure" speaks volumes to me, never ever have I seen something similar in he field of physics and astrophysics, despite there being bitter controversies there too.

As I see it, it all boils down to how the IoM has weighed up the existing results in the literature available to them in 2010. As they themselves write in the report itself, in later publications in journals and in interviews, they took a conservative attitude for admitting evidence for positive health effects other than bone health, while setting the bar a lot lower for possible negative health effects.

Once you take into account this fact, then there isn't much of a conradiction between that Lancet article saying that there is evidence for positive health effects but that there ae still uncertainties and the IoM saying that as of yet, we can't say that there are positive health effects for vitamin D beyond bone health. A statement from the IoM that such effects are now established fact would, after all, amount to a verdict that that matter has now been settled. Count Iblis (talk) 16:16, 6 November 2011 (UTC)

We're getting into specifics here. My point was merely that if the IoM report was gold quality science then we'd still see howls of criticism. There may be genuine scientific issues with the report, which we're not going to go into here. If, in the last year, there has been newer science overturning the earlier report then that may be a reason to move on. Or are these newer reviews merely written in response and coming up with different conclusions based on more or less the same science? I'd expect for such a well documented issue that we'd find non-wikipedian's saying that new science overturns that report and also potentially the report's author's admitting that their findings are now out-of-date -- yet a brief look showed the authors standing by their work. I could be wrong about these things as I haven't the time to investigate. You seem to be saying that these two sources are just taking a different angle when interpreting the science (one conservative, one not). It is not really for us to say which approach is right. If at all possible, find independent third party non-wikipedian opinions as to how to weigh these two against each other. I think you may be straying into original research trying to decide yourselves which one is more right. Colin°^Talk 16:36, 6 November 2011 (UTC)

Colin is right: the sources are not "better" or "worse"; they are merely different, and both are excellent sources as far as Wikipedia is concerned.

Given the typically slow pace of change to scientific opinion (as held by actual experts, who know the limitations of all this work), it is a little strange to say in 2011 that a 2009 or 2010 paper is entirely out of date. I would use both for now, emphasizing the (fairly small areas of) disagreement: the IOM says it's not absolutely proven, and the Lancet article says that evidence is increasing (but still lower than the evidence for bone health).

This sort of approach is better than looking me into my crystal ball to decide which major source is going to be proven right during the next decade.

(All of this makes me wonder what the MS rates are in South Asia, where Vitamin D levels are significantly lower than in North America and Europe, despite greater exposure to sunlight. If South Asians, despite low Vitamin D levels, have less MS than white people, then perhaps we'll discover that Vitamin D is only a biomarker for sun exposure, and that the real mechanism involves some other pathway.) WhatamIdoing (talk) 17:31, 6 November 2011 (UTC)

I agree that both conclusions of both large reviews should be sumarized, even if they don't agree (though to me the difference is minor).

BTW, I can't resist a comment above about the "size of the supplement industry." If you're talking about the gigantic vitamin D supplement industry, bite your tongue. Do you ever see vitamin D adverts on TV? What about the size of the bisphosphonate industry? And yet, has anybody ever had their jaw fall apart from 4000 IU of vitamin D per day? Here we doctors sit, with the best meta analysis evidence from prospective trials that vitamin D supplements do indeed treat osteoporosis, and that doses >400 IU are clearly better than 400 IU [21] and yet we have no large prospective trials of 4000 IU because there's no pharm company behind them (a few governments are doing them now, in progress, but very late-- they could and should have been done decades ago). 4000 IU of vitamin D a day is safe, and so cheap (10 cents a day) that it's almost not worth talking about. So what happened to the research? Well, the answer is clear: it didn't get done because there was no profit in doing it, and NIH and gov funded research fell down on the "job," as usual, which was (or should be) that of looking into treatments that private industry is not going to test themselves. Without good data from studies, good and honest scientists have nothing to analyze, so they write papers saying that there isn't enough evidence yet! And there's the usual resistance to writing nutritional guidelines in which most elderly (certainly anybody with osteoporosis) come out deficient. So here we are. In the same ugly place. But don't make it worse by blaming the vitamin industry (such as it is). There're lots of places and people to blame for this sorry state, but the vitamin industy is far, far down on the list. S B H arris 18:48, 6 November 2011 (UTC)

"Neutrality requires that each article or other page in the mainspace fairly represents all significant viewpoints that have been published by reliable sources, in proportion to the prominence of each viewpoint.". Due and undue weight. That's all I want. Can the contentions in reviews such as the Lancet one be mentioned in the article. Of course, but they must not be presented as a dominant or countervailing view. The IoM report carries more weight in the scientific community.

In my edits the authoritative nature or the IoM report was briefly explained then a summary of its conclusions was given prominence in the article. Lower down in the article we mentioned the other opinions reffed to reviews. They are dissenting views and to present them as being equal is giving them undue weight. DJ is giving selected reviews undue weight.

The article now has another ref "Pierrot-Deseilligny, C; Souberbielle, JC (2011 Apr). "Widespread vitamin D insufficiency: A new challenge for primary prevention, with particular reference to multiple sclerosis.". Presse medicale (Paris, France : 1983)" [1983?] DocJames is editing as if the IoM report is just another paper (he actually calls it 'a paper') and seems to think that piling up reviews will outweigh it. WP can't decide who is right between scientists but NPOV reqires giving proper weight to a 1000+ page report requested by the US government which was the work of 14 professors

On one hand we have a collage of selected review studies each led by a professor, on the other we have a report requested by the US and Canadian governments which was written by a committee of 14 professors from the premier medical body in the country (world probably). Who is right is not at issue, what is at issue is the weight that should be given to the IoM report in the article. Overagainst (talk) 19:11, 6 November 2011 (UTC)

No, clearly the situation above is going too far. In an WP article like this, there is only room for meta analyses, and preferably those done by many people, preferrably multi-institutionally, not lone people with axes to grind.

Having said that, there ought to be room somwhere for recognising that the "standard scientific view" in medicine is often very heavily influenced by pharmaceutical industry spending. Pharm sales are about $300 billion/year in the US, vs. total supplement sales, which total maybe $20 billion a year, so one industry outadvertises the other by at least 15 to 1 (in reality it's far more than that, since most supplements aren't patented, so advertising for them gets diluted and usually doesn't happen at large scales). And the research support ratio is even higher. All of this produces bias. WP, for example, tends to have longer articles on on-patent drugs than it does on off-patent drugs. Why does sertraline/Zoloft (only recently off patent) have twice as long an article as the groundbreaking fluoxitine/Prozac? Is it any better as a drug? No. So who writes these things? S B H arris 19:41, 6 November 2011 (UTC)

Please remember this is the talk page for discussing the guideline. Unless there's some clarification needed to the guideline, that would help in this case and others, then this discussion should continue in the article page. If you want additional participants, post at WT:MED. Colin°^Talk 22:55, 6 November 2011 (UTC)

While I agree with Overagainst that the IoM report is not just any other review article, we can't give this more weight than other review articles. What matters is what the IoM really did when it evaluated the existing literature and that can be found in the report itself. You can read a bit about this in this Nature article. What you read here is that the IoM didn't go about reviewing the literature like we normally would expect normal review articles to do. They took issue with some published results and they re-did the analysis in these papers arriving at different conclusions than the authors of these papers. The authors of these papers have later said that they didn't agree with the IoM's re-analysis of their results. Count Iblis (talk) 23:26, 6 November 2011 (UTC)

Colin, you say - "Both sources are "ideal". [..]The only thing we can say is that MEDRS thinks these sources are great". If they are equally, or nearly equally. great then multiple reviews count for more than The IoM report.

From the project page's 'Summarize scientific consensus'- "Wikipedia policies on the neutral point of view and not using original research demand that we present the prevailing medical or scientific consensus, which can be found in recent, authoritative review articles or textbooks and some forms of monographs. Although significant-minority views are welcome in Wikipedia, such views must be presented in the context of their acceptance by experts in the field." That is what I was getting at about tertiary sources,. Tertiary may be the wrong word for it but there is a level which is more authoritative than reviews. The IoMs report is a form of monograph which the project page says is more authorative. Project page 'books' section -"Medical textbooks published by academic publishers are often excellent secondary sources. If a textbook is intended for students, it may not be as thorough as a monograph or chapter in a textbook intended for professionals or postgraduates." The IoM report is a thorough monograph on the subject from the Institute of Medicine , the go-to body for medical matters. The IoM report comes into a category the 'Identifying reliable sources (medicine)' page talks about: "textbook intended for professionals". It is the source for information on the subject. Colin you said MEDRS thinks they're both great. If you continue to maintain that then let me use this talk page for its intended purpose and suggest changes to the MEDRS project page. I suggest that you take out all references to thorough monographs or authoritative textbooks intended for professionals and just talk about secondary sources as review articles and maybe chapters in textbooks intended for students .Overagainst (talk) 12:09, 7 November 2011 (UTC)

That's not going to happen. Science is not static, but individual publications are. The IOM report cites only works prior to 2009 in its discussion of MS. No matter how authoritative or comprehensive it was at the time of publication, until such time as the IOM updates its report it will not reflect other works published since then. More recent MEDRS are continuing to be published. That is why we routinely check Pubmed for reviews: to establish the recent direction of scientific consensus. LeadSongDog come howl! 14:51, 7 November 2011 (UTC)

Overagainst, I don't understand your suggestion. You quote a sentence that says these three sources (of the "recent, authoritative" variety):

review articles
textbooks
some forms of monographs

are identically wonderful. Then you say that you want to use a good monograph in the article. So you propose removing the endorsement of monographs from the list of ideal sources.

Why would you want to remove the guideline's support for the kind of source you want to use? WhatamIdoing (talk) 16:03, 7 November 2011 (UTC)

(EC with WhatamIdoing) Overagainst, I (also!) don't understand what you are suggesting. I'd classify the IoM report as more of a "medical guidelines or position statements published by major health organizations" than a monograph for professionals but it is that too. As I said at the start, I think such a document has more weight/authority than a contemporary review by a few authors from one institution. Do we need to update MEDRS so people can gauge the authority of such a publication, or is that something people can work out for themselves? If the two sources aren't contemporary then we need to work out if the newer source is really benefiting from new research or is merely another POV published later. Is this not obvious stuff? I'm getting the feeling folk are wanting MEDRS to resolve a real-world dispute and would like it to say the IoM report is worth 7 and the Lancet review worth 5.5 or some such simplistic weighting. The IoM document is a thousand pages long; there must be good bits and bad bits. Colin°^Talk 16:10, 7 November 2011 (UTC)

No one is suggesting using random pages from the 1000plus page report in the article. I want to use the summary, the best bits. Like these two bits "Despite the many claims of benefit surrounding vitamin D in particular, the evidence did not support a basis for a causal relationship between vitamin D and many of the numerous health outcomes purported to be affected by vitamin D intake. Although the current interest in vitamin D as a nutrient with broad and expanded benefits is understandable, it is not supported by the available evidence. The established function of vitamin D remains that of ensuring bone health, for which causal evidence across the life stages exists and has grown since the 1997 DRIs were established (IOM, 1997). The conclusion that there is not sufficient evidence to establish a relationship between vitamin D and health outcomes other than bone health does not mean that future research will not reveal a compelling relationship between vitamin D and another health outcome. The question is open as to whether other relationships may be revealed in the future." There is no benefit to supplementing according to the IoM

" Although ensuring adequacy is important, there is now an emerging issue of excess vitamin D intakes. A congruence of diverse data on health outcomes ranging from all-cause mortality to cardiovascular risk suggests that adverse health outcomes may be associated with vitamin D intakes that are much lower than those classically associated with hypervitaminosis D and that appear to occur at serum 25OHD levels achievable through current levels of supplement use" There are indications that it may be risky. These important statements should be given due weight. Overagainst (talk) 17:47, 7 November 2011 (UTC)

IOM reports are excellent sources. Recent Lancet review articles are excellent sources. If excellent sources disagree, then we should be able to briefly summarize their areas of disagreement. I'm not quite grasping the core of this specific content dispute, but would echo others' requests that the specific content in question be discussed at Talk:Vitamin D rather than here. MastCell ^Talk 19:47, 7 November 2011 (UTC)

No one is suggesting using random pages from the 1000 plus page report in the article. I want to use bits of the the summary, the best bits. Like these two bits "Despite the many claims of benefit surrounding vitamin D in particular, the evidence did not support a basis for a causal relationship between vitamin D and many of the numerous health outcomes purported to be affected by vitamin D intake. Although the current interest in vitamin D as a nutrient with broad and expanded benefits is understandable, it is not supported by the available evidence. The established function of vitamin D remains that of ensuring bone health, for which causal evidence across the life stages exists and has grown since the 1997 DRIs were established (IOM, 1997). The conclusion that there is not sufficient evidence to establish a relationship between vitamin D and health outcomes other than bone health does not mean that future research will not reveal a compelling relationship between vitamin D and another health outcome. The question is open as to whether other relationships may be revealed in the future." There is no benefit to supplementing according to the IoM

" Although ensuring adequacy is important, there is now an emerging issue of excess vitamin D intakes. A congruence of diverse data on health outcomes ranging from all-cause mortality to cardiovascular risk suggests that adverse health outcomes may be associated with vitamin D intakes that are much lower than those classically associated with hypervitaminosis D and that appear to occur at serum 25OHD levels achievable through current levels of supplement use" There are indications that it may be risky. These important statements should be given due weight. Overagainst (talk) 17:47, 7 November 2011 (UTC)

I'm not asking for an opinion on the above statements. The problem is that they can't be given due weight unless they can be correctly identified as scientific work which is at a level which is more authoritative than reviews (they are). "If the two sources aren't contemporary then we need to work out if the newer source is really benefiting from new research or is merely another POV published later." Well the IOM report was a long time coming and it has a publishing date of 2011. If there are reviews that are more recent it's for similar reasons to why primary sources will be more recent than systemic reviews. That doesn't mean that recent primary studies trump reviews, nor should it.

The nature of the "real-world dispute" is due and undue weight and that can't be assigned until the source has been identified as a primary, a secondary or a level above secondary, 'higher secondary' if you like.. Identifying reliable sources involves assessing due weight but unfortunately there seems to be a reluctance to go beyond assigning anything but secondary status, even to huge books produced by a panel of experts at the request of the US government for the specific purpose of providing authoritative conclusions about a scientific controversy. Talking about the IoM report being weighted at 7 and the Lancet review at 5.5 is quite revealing, and astounding. The weight carried by the IoM report is an order of magnitude greater at least. The problem with Wikipedia is there is not enough protection for experts as all editors are on an equality. But not all secondary sources are equal. Any more than a Phd. is equal to a full professor.

"Why would you want to remove the guideline's support for the kind of source you want to use?" Because the guideline is a waste of space unless people are ready willing and able to apply it. If it can't identify the most reliable source between two 'secondary sources', one a whopping great book by the go-to authority for medical information that was commissioned by the US /Canadian govenments to clarify the issue, the other a run of the mill systemic review, what is the use? Overagainst (talk) 20:24, 7 November 2011 (UTC)

I dunno: What's the use of telling people to stop smoking, since some of them won't? What's the use of recommending that people use seat belts every time they get in a motor vehicle, since some of them won't? What's the use of telling people not to overdose on Vitamin D, since some of them are determined to end up with hypervitaminosis anyway?

Our job here is to give the best advice that we can, not to give only the advice that people instantly obey. WhatamIdoing (talk) 05:18, 8 November 2011 (UTC)

Overagainst, the fact that you took my plucked-from-thin-air figures of 7 and 5.5 and argued about them, "is quite revealing, and astounding". [ How do you know my scale isn't logarithmic :-) ] My whole point is that numerical scoring would be a nice way of solving this dispute between editors but totally flawed. You are expecting a general medical sourcing guideline to resolve a specific dispute between experts that has spilled over to become a dispute between WP editors. Colin°^Talk 08:50, 8 November 2011 (UTC)

I was wrong to pick up on your scoring analogy, sorry. Something that may be of interest Dietary Reference Intakes for Calcium and Vitamin D (2011),page 127-

"A key component of systematic reviews of scientific literature is a specification of the quality of the available data. The AHRQ grading system is summarized in Box 4-1. In the case of the systematic analysis carried out by AHRQ-Ottawa, the Jadad scale (Jadad et al., 1996) was used for quality assessments of randomized controlled trials (RCTs). The Jadad scale is a validated scale designed to assess the methods used to generate random assignments and double blinding. The scale also scores whether there is a description of dropouts and withdrawals by intervention group. Jadad scores range from 1 to 5, and a total score of 3 and above indicates studies of higher quality. Further, to assess the quality of the observational studies, a grading system adapted from R. P. Harris et al. (2001) was used. In the case of the AHRQ-Tufts analysis, a three-category grading system (“A,” “B,” or “C”) was adapted from the AHRQ Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews (AHRQ, 2007). This system defines a generic grading system that is applicable to each type of study design including interventional and observational studies; it is summarized in Box 4-1.

The committee’s literature search identified relevant evidence outside the scope of, or not included in, the two AHRQ reports as well as newer data available after the cutoff date of the AHRQ-Tufts analysis in 2009. The nature of the literature search is outlined in Appendix E. The literature base that was included in the 1997 report of the IOM committee tasked with DRI development for calcium and vitamin D (IOM, 1997) was also considered. Additionally, information gathered as part of a public workshop and several open committee sessions (see Appendix J) and a white paper requested by the committee (Towler, 2009) were taken into account."

I'll stand by the point that not everything above the level of a primary study is remotely comparable and that the the set of set of criteria on the 'Identifying reliable sources (medicine)' page is not what it should be. Quote: "Medical and scientific organizations Statements and information from reputable major medical and scientific bodies may be valuable encyclopedic sources. These bodies include the U.S. National Academies (including the Institute of Medicine and the National Academy of Sciences), the British National Health Service, the U.S. National Institutes of Health and Centers for Disease Control and Prevention, and the World Health Organization. The reliability of these sources range from formal scientific reports, which can be the equal of the best reviews published in medical journals". Can be the equal of the best reviews? That seems to imply that the report is of lesser status than a review. That is just wrong.

Take a look at the Dietary Reference Intakes for Calcium and Vitamin D:Literature Search Strategy. What did they use in their work on the report? Studies such as the "Agency for Healthcare Research and Quality (AHRQ) evidence report issued in 2007 (Cranney et al., 2007), and the Agency for Healthcare Research and Quality evidence report issued in 2009 (Chung et al., 2009) link AHRQ Vitamin D and Calcium: Systematic Review of Health Outcomes these are true meta studies far superior to a Lancet paper and the IoM report is more superior still.

I'm all for summarizing areas of disagreement between secondary studies in articles, and I've tried to do that in my edits, but their respective views must each be given the proper weight. This is part of identifying reliable sources. It's not 'deciding which group of experts is right', just identifying authoritativeness in the medical community. Who is most reliable in other words.

WhatamIdoing, Overdosing on Vitamin D is easier than you may think Dietary Reference Intakes for Calcium and Vitamin D (2011) p.436"large-scale pooled analysis (n = 2,285) found a statistically significant two-fold increased risk for pancreatic cancer in participants withserum 25OHD levels at or above 100 nmol/L compared with those with levels between 50 to 75 nmol". 100 nmol/L = 40ng/ml. (ie less 40 to 80 ng/mL which some say is the optimum range) Overagainst (talk) 17:33, 8 November 2011 (UTC)

Overagainst, are you proposing an edit to wp:MEDRS? If not, please take the discussion to talk:Vitamin D where it belongs. LeadSongDog come howl! 19:55, 8 November 2011 (UTC)

Medical and scientific organizations section

(continuing from above) I now think Overagainst has found a problem with the current MEDRS wording. The section "Medical and scientific organizations" was added by Eubulides in response to a discussion at Wikipedia talk:Identifying reliable sources (medicine)/Archive 3#Expansion: CDC's website?. Overagainst rightly questions the "can be the equal of the best reviews published in medical journals" as implying they can be no better. Additionally, I don't agree with the wording "but are generally less authoritative than the underlying medical literature". I think the latter applies to the lay websites and literature produced by the CDC and the NHS, but not to reports such as the above NIH study. I've generally considered the full guidelines produced by NICE and SIGN to be about the best source one could wish to use. There is an extensive review process involved in those guidelines that IMO is superior to mere peer review for journals. I'm less familiar with the equivalent guidelines for other countries. I question the idea that "medical literature" is only that published in academic journals.

When writing Ketogenic diet I found this source:

Kossoff EH, Zupec-Kania BA, Amark PE, Ballaban-Gil KR, Bergqvist AG, Blackford R, et al. Optimal clinical management of children receiving the ketogenic diet: recommendations of the International Ketogenic Diet Study Group. Epilepsia. 2009 Feb;50(2):304–17. doi:10.1111/j.1528-1167.2008.01765.x. PMID 18823325

to be my favourite and almost like it was written for Wikipedia. It was, like Overagainst would like me to say, an order of magnitude more useful for our purposes than any old review by any old expert in any old journal. It was commissioned by a charity and "Subsequently endorsed by the Practice Committee of the Child Neurology Society". It isn't a classic journal review. I'd be interested to know what the Wikidocs think about these various non-review reports, as many folk here have much more knowledge/experience than me. I think this section should be revised. Can I please request that in this sub-section, the words "Vitamin D" are banned! Colin°^Talk 20:11, 8 November 2011 (UTC)

Hmmm, I wouldn't have taken that contrarian meaning from "can be the equal of", but I take your point that some editors might do so. It could be mistaken as wp:weasel worded anyhow, which is not at all characteristic of Eubulides' fine work. Still, if he had chosen "can be equal to, better than or worse than" the semantic value would be nil. Worse, he would have been taken as having falsely implied that there is no reason to prefer one over the other. The rare few shining exceptions are reasons to IAR, not to change a very useful rule that keeps out lots of lower-reliability sources when good ones are available. Can you think of alternate wording which would be an improvement? LeadSongDog come howl! 20:47, 8 November 2011 (UTC)

I'm more worried about the "generally less authoritative than the underlying medical literature" statement, but want to hear from others. Colin°^Talk 22:09, 8 November 2011 (UTC)

I would remove the "generally less authoritative" clause that you're concerned about. I agree that evidence summaries, guidelines, and reports from major, reputable medical and scientific bodies are among the very best secondary sources we can ask for. Whether such expert-body statements are "better" than a given review article is certainly a case-by-case determination, so we shouldn't enshrine the idea that expert-body statements are generally less valuable across the board. MastCell ^Talk 22:16, 8 November 2011 (UTC)

(The 2011 book Ketogenic Diets: Treatments for Epilepsy and Other Disorders mentions the need for calcium supplements. It refs the IoM report on calcium HERE). I would say that that book is representative in that they choose to take note of the IoM's recomendations over other authorities or experts. Surely a rule to keep lower-reliability sources out could mention the IoM and say its reports and statements on medical matters are considered highly reliable.Overagainst (talk) 12:59, 9 November 2011 (UTC)

I'm not so sure that we have a real problem here. Here's the full sentence:

The reliability of these sources range from formal scientific reports, which can be the equal of the best reviews published in medical journals, through public guides and service announcements, which have the advantage of being freely readable, but are generally less authoritative than the underlying medical literature.

Okay: Who's prepared to argue that a patient-oriented summary written so that the average 12 year old can understand it ("public guide") or a press release ("service announcement") is at least as authoritative as the underlying medical literature? Anyone?

I'm not. The "formal scientific reports" can certainly equal the best reviews (NB: not just any old review), and they are excellent, authoritative sources, but the public relations stuff is less authoritative than the formal literature. (Perfectly usable in some cases, but still less authoritative.) WhatamIdoing (talk) 22:06, 10 November 2011 (UTC)

You know, on reading it again, you are right. There's nothing wrong with it. I must have parsed the sentence wrongly. Colin°^Talk 22:17, 10 November 2011 (UTC)

And if you can misread it, then anyone can. I'm open to changes to the presentation. Do you think it might help if these were in separate sentences, like "Reports can be the equal... However, patient-oriented guides...."? WhatamIdoing (talk) 22:24, 10 November 2011 (UTC)

"Statements and information from reputable major medical and scientific bodies may be valuable encyclopedic sources. These bodies include the U.S. National Academies (including the Institute of Medicine and the National Academy of Sciences), the British National Health Service, the U.S. National Institutes of Health and Centers for Disease Control and Prevention, and the World Health Organization. The reliability of these sources range from formal scientific reports, which can be the equal of the best reviews published in medical journals, through public guides and service announcements, which have the advantage of being freely readable, but are generally less authoritative than the but are generally less authoritative than the underlying medical literature".

How much weight to give a report depends on the source of the report I'm prepared to argue that a patient-oriented summary written so that the average 12 year old can understand it ("public guide") or a press release is in the same category as as say a 1000 page report on particular issues by a committe of 14 experts from the (IoM) premier medical authority (in the world probably) requested by the US government to resolve a debate about those medical issues, if both are from the IoM. One is written at a much higher level and is suitable as a source but it is not necessarily more reliable than a simple statement, from the IoM. Main point is the suitability of formal scientific reports for use as a source on WP depends on the the source of the report (and whether it is a major report) but it's quite wrong to imply that such reports can do no better than equal the best reviews. And it is not clear whether it's meant that a major report should at best be given as much (but never given more) weight than a single excellent review (obviously wrong IMO) or multiple reviews. It seems it is being suggested that several reviews from prestigious journals can and should outweigh the biggest of reports from the best source. The 'Medical and scientific organizations' section is trying to do too much in too few words, it should be expanded and make clearer. It should be explicitly stated that the formal scientific reports produced by the Institute of Medicine are a reliable source for medical information. It would be even better if the IoM was stated to be the most reliable source for medicine, otherwise it is open to construct a collage of review for a viewpoint. Multiple review sources don't aggregate into a superior source that outweighs a major report from the IoM. Overagainst (talk) 10:16, 14 November 2011 (UTC)

The trouble is, we can't say that an IoM report always trumps any review forever. I would agree with you that a "can do no better than equal the best reviews" interpretation is wrong and we should reword to avoid that meaning. Perhaps we need to document ways of examining the authority of a report/review (in the "Assess evidence quality" section or similar). For example, whether the report/review includes many experts from many institutions. Does it draw experts from various fields and seek to establish consensus? Are the conclusions arrived at systematically or informally using the author's own judgement? How is the report/review itself reviewed prior to publication -- a normal literature review would undergo a very different set of checks from a SIGN report, say. Could the purpose and authorship of the review/report influence the conclusions (e.g., conservative vs optimistic). How was the review/report received? And has there been significant new research since the review/report was conducted? Colin°^Talk 13:05, 14 November 2011 (UTC)

Superceeded IoM reports don't trump reviews forever, but the IoM keeps up to date, if there is a need they commission a new report. 'Reception' is a rather subjective criterion but one way of examining the authority of a report/review is how are they recieved. IoM reports always get a very respectful hearing as far as I know. When Kossoff was working on the keto diet he reached for the report from Institute of Medicine of the National Academies. If you are treating children with a novel therapy you want the most reliable sources you can get. Overagainst (talk) 13:32, 14 November 2011 (UTC)

I've no idea if your faith in the up-to-dateness of IoM reports is valid. I can't find anything useful on their site about the management of epilepsy, say, so they aren't the answer to everything. NICE and SIGN guidelines are updated too but, for example, their epilepsy guidelines are dated 2004 and 2005 respectively. This paper is an interesting discussion on comparing epilepsy drugs. We all like evidence-based-medicine, but sometimes the studies haven't been done, or the evidence is weak. If I was writing an article on epilepsy drugs, I'd consult those guidelines for sure but I'd be very aware of how old they were. I'd try to find as many recent high-quality sources as I can and hopefully find they all tend to agree. In the dispute that caused this mess (which shall not be named) there's obviously disagreement, which makes our job much harder. Colin°^Talk 15:04, 14 November 2011 (UTC)

The IoM move carefully and cautiously and want plenty of data, including data on long term effects, to work with before reaching any conclusion. While they are not the best source on relatively novel therapies which may be one of the things that they have not got round to issuing recommendations or reports on, when they issue a major report it's usually because they, or someone like the US government, perceives a need to clarify things. Obviously there is going to be some negative reaction from those soi disant experts who've given opinions that are not borne out by such a report. In the aftermath of a major report by the IoM the mainsteam medical people incuding specialists in their own areas (like keto diets) give that report more weight that any other source. For example the keto doc wanted the most reliable information available about calcium requirements appropriate for children - he reached for the recent IoM report on Calcium. Recent, and relatively recent, major IoM reports are the gold standard. Overagainst (talk) 11:30, 15 November 2011 (UTC)

I'm glad to see some agreement that even the best of scientific sourcing is perishable. The process by which such bodies maintain currency is not always as obvious as the results. It might be worth some discussion as to how editors should, when such conflicting evidence is seen, look for signs that the body behind the formerly authoritative report might be in the process of updating it. Usually there are expert comittees engaged that include some or all of the previous version's authors, for example. Shedding light on how these bodies manage their review workload could be helpful.LeadSongDog come howl! 16:31, 15 November 2011 (UTC)

Overagainst, I think you're wrong about the patient leaflets being just as authoritative as a full report. The public guides are often seriously incomplete. To give one example, a decade ago, the patient guides from these agencies said American women over the age of 40 ought to consider annual screening mammograms, and every woman over the age of 50 ought to get a mammogram every single year. The advice has been materially changed since then, but this was false even according to the then-current recommendations. The actual recommendations at the time gave different advice to women with BRCA mutations and other hereditary breast-ovarian cancer syndromes, to women who had previously had breast cancer, to women whose life expectancy was less than ten years, to women whose health was so poor that treatment would have been inadvisable, etc. So the public guides, with their one-size-fits-all oversimplification, were noticeably worse sources than any half-decent review on the subject would have been. An editor who relied on the oversimplified public advice would have produced a far worse article than one who relied on a current review, textbook, or full report. WhatamIdoing (talk) 18:47, 15 November 2011 (UTC)

(The IoM had a report on screening out in 2004). Regarding patient guides it's not clear if your talking about IoM advice. I would say that the source is the most important factor what the IoM gives out is reliable because of the source it comes from. Where there are relatively recent IoM reports available those be given most weight because the come from the IoM.Overagainst (talk) 19:12, 15 November 2011 (UTC)

Arbitrary divide to keep above section vitamin D free

Reports from institutions that have an advisory role used for policy making are generally less reliable than the average scientific review article. This is particularly true in medicine. An institution like the IOM or the WHO will analyze the existing scientific results from a public health perspective. This leads to a lot of weight given to tentative evidence that some effect may be dangerous. So, despite the old model abut health effects of exposure to radioactivity being debunked for a long time and this spectacularly demonstrated after the Chernobyl disaster (the hundreds of thousands of extra cancer cases that never materialized), the WHO still uses this old debunked model to estimate health effects of exposure to radiactivity.

Another example is the recent WHO report on exposure to EM radiation from mobile phones. Despite adverse health effects being shown to not exist by almost all of the available scientific evidence and being theoretically implausible, the report says that exposure to EM radiation from mobiles may not be safe.

Final example. Old IOM reports on Vitamin D say that vitamin D in doses that are now considered to be safe without question, can cause birth defects. But this was not based on what at the time was considered to be good evidence that turned out to be wrong later. see here:

"As doctor Hollis explains: “Vitamin D supplementation during pregnancy remains controversial largely due to severe misconceptions about the potential harm it may cause to the fetus. Surprisingly the scientific debate has made little progress since Dr. Gilbert Forbes made a recommendation of 200 IU per day in 1963, was based on a hunch”. Researchers conclude that higher dosages of vitamin D supplementation are safe and effective for expecting women." Count Iblis (talk) 16:30, 11 November 2011 (UTC)

You say "Reports from institutions that have an advisory role used for policy making are generally less reliable than the average scientific review article." Do you have any evidence for this? As the saying goes, "the plural of anecdote is not data". My gut feeling and limited experience suggests this is bollocks. Colin°^Talk 18:26, 11 November 2011 (UTC)

How about some data the disagrees with Count Iblis' position? PMID 20693976, an "average review", says that cell phones might cause brain cancer. So does PMID 20429163, another "average review". That's two out of the first three reviews in the list if you search PubMed for "cancer cell phone". Since the WHO's report and these "average reviews" agree, why are we trying to say that either is worse than the other?

I'd actually prefer the WHO report, because it explains the limitations of the evidence, e.g., that the evidence that cell phone use causes cancer is just as strong (or weak, depending on your perspective) as the evidence that drinking coffee causes cancer. WhatamIdoing (talk) 19:55, 11 November 2011 (UTC)

MEDRS and research sections

WP:MEDRS is being used to justify removing research sections that describe the drug pipeline for diseases, e.g. this and this. Do contributors here think that this material is undesirable in Wikipedia? I think secondary sources are preferable but when it comes the research pipeline, they are rare and usually out-of-date. Should there be an exception to requiring secondary sources or does this material not belong in Wikipedia? pgr94 (talk) 17:43, 27 November 2011 (UTC)

Have bee recently working on the topic anaphylaxis and I have found review article to support the content of the research section. --Doc James (talk · contribs · email) 18:08, 27 November 2011 (UTC)

Similarly, I had no problem finding secondary review articles that discussed current research efforts in Tourette syndrome. SandyGeorgia (Talk) 16:27, 28 November 2011 (UTC)

Any studies or research that deserves WP:WEIGHT should be in found in secondary sources such as reviews. Yobol (talk) 01:08, 28 November 2011 (UTC)

To elaborate on Yobol's point, we pseudonymous editors of WP are unable to verifiably assess which published primary sources deserve weight in an article, so we must rely on published reliable sources to establish that a specific primary source is notable. By definition, such publication is secondary. Without it, we are engaged in original research when we assign weight to a primary source. LeadSongDog come howl! 16:24, 28 November 2011 (UTC)

I completely agree that the significance of different types of treatment being researched should come from a secondary source. To clarify my query: is a pipeline section appropriate? And if it is appropriate where should it be sourced from? Review articles are inevitably going to be more out of date than clinicaltrials.gov Also, what kind of source is clinicaltrials.gov seen as? primary/secondary? pgr94 (talk) 16:51, 28 November 2011 (UTC)

Yes it is one way that we attempt to deal with recentism and the fact that Wikipedia is an encyclopedia not news source. We want knowledge to pass a small test of time before it is included.--Doc James (talk · contribs · email) 15:02, 7 December 2011 (UTC)

RSN

Another editor suggested that people watching this page might like to join the discussion at Wikipedia:Reliable sources/Noticeboard#Paraphilic_infantilism. This involves a subject that is rarely discussed in scholarly sources and for which no "mainstream" view on major subtopics (like the cause) really exists. So the article is mostly "one expert said this, another said that, here's another view".

The dispute is apparently over whether a peer-reviewed paper by an expert is a reliable source for what that expert's own theory is. This dispute might be driven in part by editors' squeamishness over the expert's use of the term "autoerotic pedophilia" (which means an attraction to the idea of one's own self being a child, not an attraction to other children). WhatamIdoing (talk) 21:43, 6 December 2011 (UTC)

Actually, even one of the involved experts/authors has used the DSM as a mainstream, reliable, consensus definition for infantilism[22]. It lists infantilism clearly and simply under the heading of masochism. This contradicts the fringe view that is being edit warred for. WAID and WLU don't wish to accept this (and the prevailing view of that discussion, apparently). It seems plainly clear this fringe theory, being promoted only by one facility and one editor here, lacks the independent sources required by the fringe theories guideline.

Of course, the presence of a widely verifiable source - the DSM - that contradicts the fringe theory would render this discussion moot. The fringe theories guideline is quite clear.

WAID, you should have presented both sides of the debate in your invitation, to avoid the appearance of further votestacking. I also notice that this was taken up at RSN, where you and WLU were so successful in the past. Then, the effort was to remove almost all references to the DSM, to make way for many citations to few papers by the one facility promoting this fringe view; this fringe view that contradicts the DSM. BitterGrey (talk) 07:01, 7 December 2011 (UTC)

What the page (paraphilic infantilism) really needs is the attention of editors willing to look into the sources with critical eyes, experience in wikipedia's policies and guidelines, and engage on the page over the long term. There are some subtle, nuanced issues that need to be addressed and right now with only two editors, Bittergrey and myself, active on the page, the only thing happening is disagreement and repetition. WLU (t) (c) Wikipedia's rules:^simple/_complex 14:44, 7 December 2011 (UTC)

Based on the response at RSN, it seems only WLU and WAID consider "subtle, nuanced issues" present. Most seem to agree that texts written by coworkers aren't independent, and thus per the fringe view policy, the fringe view that WLU is pushing should not be included at all, much less in multiple locations. I have repeatedly quoted Wikipedia policy in my attempts at discussion. WLU has not. BitterGrey (talk) 02:36, 8 December 2011 (UTC)

Based on the response at RSN, I haven't seen anyone agree with you, and I've seen multiple editors tell you that Freund and Blanchard's peer-reviewed papers are unquestionably reliable sources for what Fruend and Blanchard said. You're not managing to find anyone to agree with you over at FRINGE, either. WhatamIdoing (talk) 21:47, 19 December 2011 (UTC)

The value of peer-reviewed studies reflecting developments since the last review is greatly overlooked.

The following sentence appears in the policy article but is not emphasized.

If an important scientific finding is so new that no reliable reviews have been published on the subject, it may be helpful to cite the primary source that reported the original result.

Wikipedia should emphasize reviews when they are available, and should foster awareness of what a review is and where to find them.

Unfortunately, the above quoted sentence is so overlooked that I experience at least some Wikipedians seem to believe they are making the encyclopedia better by censoring and reverting all non-review sources, even studies that are are peer-reviewed and reputably published, and even though there are no review sources on point.

The practice of censoring good faith additions citing peer-reviewed studies that happen not to be review, where no review exists, continued on its current course, will end up with Wikipedia out of date since reviews, by definition, lag studies. Censoring by reverting good faith sections based on peer-reviewed published studies also bites the newcomers.

One of the sections above question (roughly): "What's a review and where do you find them?" reminding us that medical citation and the weighing of authorities involves a non-trival learning curve. The question is legitimate, and deserves a page answering it.

A non-biting way to handle duly footnoted good faith discussion where the footnote(s) happens not to be (a) review(s) would be to have a quick look and see if there is a review. If not, leave the section as is, if it is otherwise unobjectionable. Peer-reviewed published studies are not bad per se, and where they are the best information available, they are better than either no-footnote sections or omitting to bring forward salient studies.

If a review exists, substitute it if appropriate. If you feel you lack time to check, or feel the need to delete citations, don't click "revert;" move the section to the discussion page, cite this WP "reviews are good" section, thank the author. — Preceding unsigned comment added by Ocdnctx (talk • contribs) 21:23, 19 December 2011 (UTC)

Medicine-related articles aren't the only subject that attracts occasional overzealousness.

The basic problem is allowing Wikipedia editors to decide whether the information in a primary source is "the best information available". That tends to lead to POV pushing (e.g., people saying that they personally know that ____ is the best information available because they personally believe ____ to be true), and we want to discourage it, even if that means that Wikipedia does not report the bleeding edge of science. WhatamIdoing (talk) 23:14, 19 December 2011 (UTC)

Using multiple sources, rather than just one, may result in a more accurate text

For instance, above this language appears, in which the IOM makes unmistakably plain that it is talking about the state of current evidence, not whether vitamin D may ultimately prove related to other significant benefits:

The conclusion that there is not sufficient evidence to establish a relationship between vitamin D and health outcomes other than bone health does not mean that future research will not reveal a compelling relationship between vitamin D and another health outcome. The question is open as to whether other relationships may be revealed in the future."

Yet this section could be incautiously misread as the opposite - as a statement about benefits of Vitamin D, rather than a statement about evidence. Which is exactly how "no sufficient evidence" results are typically reported in the press.

Against this backdrop, it's hard to see how the accuracy function is served by censoring out the ongoing accumulation of peer-reviewed, reputably-published evidence, not yet deemed significant, of possible benefit (or detriment) to vitamin D. Certainly, the IOM makes no call for such censorship.

Adding additional sources, as well as careful reading and accurate reporting of the precise and restrained IOM review of evidence, may make a more balanced and accurate article. — Preceding unsigned comment added by Ocdnctx (talk • contribs) 21:48, 19 December 2011 (UTC)

Basically, Wikipedia is not a scientific news source. We're not "censoring" the latest evidence; we're only declaring that it's not the job of an encyclopedia to process or report the latest evidence. Wikipedia is an encyclopedia, not a collection of news reports. WP:There is no deadline for adding evidence, and there is definitely no deadline that requires us to rush out in front of the best types of sources (like review articles and medical reference works). WhatamIdoing (talk) 23:17, 19 December 2011 (UTC)

It is the opinion of most here that high quality secondary sources other than the IoM may also be used. Thus as long as a position is widely supported it can be added.Doc James (talk · contribs · email) 23:35, 19 December 2011 (UTC)

Should we remove this line

Wikipedia is not a new source thus I think we should remove

If an important scientific finding is so new that no reliable reviews have been published on the subject, it may be helpful to cite the primary source that reported the original result.

The problem is before a primary result is incorporated into a secondary source it is hard to determine what it means and we as editors should not be making these determination per WP:OR.Doc James (talk · contribs · email) 23:42, 19 December 2011 (UTC)

This is an important and valid point, DocJames. But WP:OR also states

Policy: Unless restricted by another policy, primary sources that have been reliably published may be used in Wikipedia, but only with care, because it is easy to misuse them (my emphasis)

And while it is true, as you say "… before a primary result is incorporated into a secondary source it is hard [my italics] to determine what it means …" it is not necessarily impossible, even for non-medical editors. We have e.g. the hierarchy of evidence. We also should consider that good primary research such as robust RCT studies (=primary research) is often accompanied by a sort of 'editorial review' in the same issue. (A case in point would for instance be PMID 21076131 and a robust review-ish comment in the same issue: PMID 21076130.) I think we may just have to live with the fact that ‘quality’ or 'reliability' in medicine has to judged on a case-to-case basis, sometimes.

So exe(m)ptions to the MEDRS rules should be allowed but

...only with care, because it is easy to misuse them...

Medicine isn't sharp-edged, it's often very fuzzy... (Sorry if come across as pontificating) Sleuth21 (talk) 15:13, 20 December 2011 (UTC)

For practical reasons, we sometimes need to acknowledge whatever it is that's just made a big splash in the news. I'd usually rather have articles cite an original scientific paper than a news story that someone is wrongly trying to claim is "secondary". WhatamIdoing (talk) 02:43, 21 December 2011 (UTC)

If an important scientific finding is so new that no reliable secondary sources are available it may be appropriate to delay coverage until such sources appear. Otherwise bring up the primary source that reported the original result on the discussion page to determine consensus for its inclusion.

This would be better.--Doc James (talk · contribs · email) 02:50, 21 December 2011 (UTC)

I'm not convinced that's true. It may be appropriate to delay... and it may not be, and having people put forward their local newspaper article on it with the erroneous but very common claim that the media source is "secondary" is not an improvement. Furthermore, if you're talking about a rare disease, the review cycle can actually be several years long. It's not all heart disease and breast cancer, after all.

Merely being a primary source is not necessarily a bad thing, either. The report on a randomized, controlled trial involving ten thousand patients is a primary source, and there's nothing wrong with us providing simple descriptions of the results while we're waiting for better sources.

We could, however, suggest both: if the finding is new, it may be appropriate to delay coverage, or it may be appropriate to cite a primary source—but carefully, because it's very easy to abuse primary sources. WhatamIdoing (talk) 05:50, 21 December 2011 (UTC)

Yes it "may" be appropriate to cite a 10,000 person RCT before a review comes out but it is not appropriate to cite an article on 40 mice or 20 petri dishes or even an RCT of 12 people in a condition that affects millions. How do we word this to address both issues?Doc James (talk · contribs · email) 07:27, 21 December 2011 (UTC)

What about robust, substantial, or particularly relevant primary studies? That would cover a small well designed robust RCT, the bigger, substantial RCTs, and even a small non-blinded study in mice (if relevant, pace early tests of penicillin). But I would maintain that "in the final analysis" this problem (which type of study do we accept as a source in WP) has to be decided on a case-by-case basis and in context. Certainly a news report in the Posemuckel Daily Herald alone should never be acceptable. Sleuth21 (talk) 10:05, 21 December 2011 (UTC) Sorry, for Posemuckel read Podmokle. Sleuth21 (talk) 10:46, 21 December 2011 (UTC)

If there are no reviews available than may be. But if there are 10 or 100s we should not be using primary research.Doc James (talk · contribs · email) 11:18, 21 December 2011 (UTC)

We seem to think that section could be improved, so I've had a go at it. If you think it could be improved further, please feel free. WhatamIdoing (talk) 17:11, 22 December 2011 (UTC)

I like the new changes.Doc James (talk · contribs · email) 01:24, 23 December 2011 (UTC)

Investigator's brochure

Where does the above type of document fit on the MEDRS hierarchy of reliability? Specifically, I'm wondering about this. It was commissioned by Multidisciplinary Association for Psychedelic Studies and written by a scientist (who works for MAPS) who has several related publications (according to my Pubmed search). In my opinion, the document is like a good review article and nicely summarizes research in the field up to its publication date (2007). However, it is not peer-reviewed, and I'm unsure about its reliability; I'd like to update the wikipedia article using this source, but desire to retain the highest standards of MEDRS-sourcing (preparation for FAC). Opinions? Sasata (talk) 05:20, 22 December 2011 (UTC)

Probably not a reliable source. The drug you hyperlink to (Psilocybin) generates 500+ articles on PubMed, none of which is a systematic review or meta-analysis. There are 11 RCTs, perhaps not the best examples of a robust, substantial, or particularly relevant primary study - quoting myself (see previous chapter above).

MAPS’ WP website is of good quality, MAPS itself looks like an institution promoting the use of psychedelic drugs, albeit on a high and ethically sound level. The website where you found the Investigator's Brochure (Scribd: a document-sharing website: ‘YouTube for documents’) conveys, of course, no quality mark.

Getting MAPS up to FAC standard may be very worthwhile: it has a fascinating history and can (at the very least) be seen as an important socio-medical phenomenon. Good luck with editing the article to FAC level.

But an Investigator's Brochure as WP:MEDRS good secondary source? No (in my opinion, obviously). Sleuth21 (talk) 08:15, 22 December 2011 (UTC)

Thanks for your reply. I think I'll just read and absorb the conclusions of the document, but won't use it as a source for the psilocybin article. Sasata (talk) 09:18, 22 December 2011 (UTC)

Good luck with developing the psilocybin WP article to FAC standard. I think you are nearly there. Sleuth21 (talk) 10:17, 22 December 2011 (UTC)

The document in question is nearly unreadable on my machine. Has it been published? Does it have a PMID / ISBN?Doc James (talk · contribs · email) 08:30, 22 December 2011 (UTC)

No, it hasn't been published in a journal, nor have an ISBN. In my (limited) understanding, this is a document prepared by the researcher to summarize the field of knowledge for whatever drug they want to investigate, and is written under "regulatory codes and guidances" devised by the FDA (in this case). Sasata (talk) 09:18, 22 December 2011 (UTC)

Oh, it's a perfectly reliable source... for extremely limited circumstances. It's self-published and any use must conform with WP:SPS rules. That pretty much means that you could use it for simple descriptions of its contents along with WP:INTEXT attribution. WhatamIdoing (talk) 16:59, 22 December 2011 (UTC)

should articles about medical conditions only deal with medical information? what about social practices?

Should articles about medical conditions only deal with medical information? Or can they also report about what people actually do with their disease, (as far as this practice meets WP:notability), even if the medical community has not written anything about it? The current guidelines WP:MEDRS do not deal with this topic: they deal with non-academic sources http://en.wikipedia.org/wiki/Wikipedia:Reliable_sources_%28medicine-related_articles%29#Other_sources as sources of medical information, but they do not deal with them as sources of information about social practices related with a medical condition. This is different: I think that medical information should clearly be separated from social practices, but I also think that both can be part of the same article. For example, evolution article has a section "Social and cultural responses".Mokotillon (talk) 14:47, 24 January 2012 (UTC)

Mokotillon, to avoid fragmenting the discussion, can we post a request once and merely link to the discussion in other places if required. Let's discuss this at WT:MED where there are likely to be more watchers. Colin°^Talk 16:26, 24 January 2012 (UTC)

Open Access Journals: Should I be asking this here instead?

I posted this inquiry about open access pay-to-play journals at the Project Medicine talkpage Wikipedia_talk:WikiProject_Medicine#Open_Access_Journals_and_related_issues, but perhaps it belongs here instead. If so, let me know and I'll move it here instead. Thanks Fladrif (talk) 16:28, 24 January 2012 (UTC)

Listing and reliability

Following a recent discussion at RSN[23], there now seems to be an assumption that a research review appearing in a new or open access journal must be listed in specific indexes (MEDLINE, ICMJE, CSE) in order to qualify as a reliable source. Is this a course of action that should be incorporated in policy? Can we determine that a research review is not high quality if not on these lists? Spicemix (talk) 20:55, 29 January 2012 (UTC)

I wouldn't call this an assumption; it looks more like a consensus. And, it would probably be a good idea to formally incorporate it into policy. Commercial publishers are churning out new medical journals literally by the dozen every single day, most on the author-pays open access model. Most will probably fail either for lack of interest or lack of scholarship, some will continue to be published but remain obscure, and a few will eventually come to be regarded as reputable and reliable. The requirement of MEDRS is that a journal must be "reputable". By definition, that means that it has gained acceptance in the medical community as reliable. If it is not indexed in the core indexes for medical publications like MEDLINE, PUBMED, ICMJE and CSE it can hardly be considered to have that acceptance - inclusion in those indexes is a very low bar indeed. Being listed is not an automatic qualification; not being listed should, however, be an automatic disqualification. If it is not widely cited by other researchers, it cannot be considered to be "reputable". If a journal is brand new and so obscure that it can't even get in those indexes, it definitely should not be used as a source on medical articles. After it is indexed, and after it is widely cited, it will then, and only then, meet the threshold of consideration as being "reputable".Fladrif (talk) 18:55, 31 January 2012 (UTC)

It could be added to the guidelines, but reading the summary by the editor whose closed the discussion, it seems that the reason the journal is not listed in the indices is that it is not a reliable source. TFD (talk) 23:58, 31 January 2012 (UTC)

A couple of thoughts:

It's not just open access journals or disreputable groups that charge for publishing. Some highly reputable journals have page fees. PNAS charges all authors $70 a page, plus other assorted fees. Circulation has a similar list of charges. Stroke charges for all publications, including letters to the editor. Blood charges $50 to even submit a paper for review. Charging for publication is not an indication of poor quality.
In terms of the "vanity press" argument, I don't see any particular difference between paying the publisher hundreds (or thousands) of dollars to publish your paper and letting them charge readers $30 a copy to read it vs paying an open access journal the same amount of money to publish your paper, in return for a promise that they won't charge the readers.
I understand that index listing takes a little while (as does acquiring a reputation), so new journals should probably not be evaluated strictly on that basis.
Some really lousy journals are listed on those indices. See, e.g., Medical Hypotheses#Abstracting_and_indexing: they don't even pretend to do peer-review.

So I don't think that we want to treat all "paid" journals the same, and I don't think that "open access" or "being indexed" is a good marker of quality.

Finally, it's worth remembering what WP:RS's FAQ page says:

"Are there sources that are "always reliable" or sources that are "always unreliable"?: No. The reliability of a source is entirely dependent on the context of the situation, and the statement it is being used to support. Some sources are generally better than others, but reliability is always contextual."

Even the journals that we say are generally "unreliable" are reliable for certain (limited) purposes, and those that we say are generally "reliable" are completely unreliable for other purposes. The issue is complicated, and simplistic rules aren't going to be helpful. WhatamIdoing (talk) 01:05, 1 February 2012 (UTC)

I agree wholeheartedly that being listed on these indices is no guarantee that a journal is a reliable source or accepted; there are indeed many journals with no peer review on these lists. But getting listed is such a low bar that not being listed on any of them has got to be a red flag that it is not yet accepted at a level that we would regard as reliable. Can anyone identify clearly reliable peer-reviewed medical journals that aren't listed on these indices? Just like being peer reviewed or not is a bright-line distinction, not being indexed on any of the leading indexes should be a bright line disqualification of a source.

Also, it is clear that the consensus at RSN and contentious complaints from advocates on the RSN talkpage and even at ANI was that Nutrition and Metabolism was not a reliable source for medical claims being made by proponents in the Transcendental Meditation research article, and not being listed on these indices was only one of many factors. Nor was being an open access publication the only or deciding factor. What User:Spicemix is not disclosing is that the same factor of appearing on none of these leading indices is one of several factors under discussion at another RSN thread involving Health Science Journal [24] One uninvolved editor has opined that it is not a reliable source for similar claims at the same article; no-one has attempted to defend the source as reliable nor to defend the use to which it is being put in the article. It seems that there is a pattern of TM research being reported in obscure and recent journals, and this question seems directed to the possibility that unindexed obscure journals will fail as sources for those seeking to push that research on Wikipedia. Fladrif (talk) 01:53, 1 February 2012 (UTC)01:48, 1 February 2012 (UTC)

You're missing the point of the FAQ: There are no bright lines. What makes a source be reliable or not depends on how you use it, not just on whether it's peer-reviewed. A non-peer-reviewed source can be perfectly reliable. This guideline recommends multiple types of non-peer-reviewed sources: practice guidelines, position statements, medical textbooks, and more. None of those typically undergo peer review.

Additionally, it's not enough to say "This is a poor publication, so everything in it is unquestionably rubbish." We sometimes accept self-published personal blogs as reliable sources. We accept business' websites as reliable sources. Surely even the worst-run academic journal isn't as completely lacking in editorial oversight as someone's personal blog. You've got to consider all the facts and circumstances. If Einstein published an article in Rubbish Journal, we'd still accept the article, because the creator of the work is sufficiently strong to outweigh our concerns about the publisher. (We'd probably wonder why he was publishing in that journal, but we'd accept it.) "Source" has three meanings on Wikipedia. "Publisher" is only one of the three. You only need one of them to be good enough.

I'm not defending the particular instances at hand; they look pretty dubious to me. But it's a bit more complicated than that. We can't just say "journals must be indexed" and expect that to weed out garbage. We can, however, reasonably predict that any statement like that will result in people saying "See, it's indexed, and therefore obviously reliable". WhatamIdoing (talk) 04:23, 1 February 2012 (UTC)

WaId makes some good points here, and that last is very persuasive. Still, it seems that absent indexing, absent a record of citations, and absent the publishing history that might eventually establish a new journal's impact, we don't have much grounds for saying that journal has a track record of responsible publication practices. A presumption against such journals should simply say the journal doesn't add credence to the authors. Effectively we would treat its contents as self-published until such time as it should establish a good reputation. LeadSongDog come howl! 06:27, 1 February 2012 (UTC)

I think that applies to any new publication, whether it claims to be an academic journal or a local newspaper. You can't have "a reputation for fact-checking and accuracy", to use the wording from WP:RS, until you have a reputation in the first place. WhatamIdoing (talk) 00:32, 2 February 2012 (UTC)

I'd like to mention that there are at least a few truly excellent journals that have embraced the open-access model. The flagship example would have to be the Public Library of Science (PLoS) family of journals. Even the lowest-impact member of this open-access, Creative-Commons-licensed seven-journal family, PLoS ONE, still pulls a respectable impact factor of 4.4. The oldest journals in the group (PLoS Biology and PLoS Medicine, started in 2003 and 2004, respectively) have impact factors of around 13. Interestingly, a few high-impact 'traditional' journals have moved to open-access; BMJ, for instance, has made all of its research articles open-access since 2008.

All of these journals are, of course, indexed in MEDLINE, and I doubt that anyone would have a problem considering them to be reliable sources by any reasonable interpretation. Nevertheless, I feel it's worthwhile to bear in mind that "open-access" doesn't have to be a dirty word, and that there are some remarkably good scientific periodicals that are published that way. TenOfAllTrades(talk) 14:53, 1 February 2012 (UTC)

Others release all articles after a set time period, making them sort of open access and sort of not. I believe that Blood makes all articles free after a year, and I've seen others that do this after five or ten years. (Actually, I don't understand why they don't all make articles free after five or ten years.) WhatamIdoing (talk) 00:32, 2 February 2012 (UTC)

Many thanks to all for these comments. They make clear just how many factors need to be considered when evaluating reliability.

RSN has indicated that Journal of Nutrition and Metabolism can't yet be considered a reliable source, so we'll have to wait and see there, but there are some interesting issues that arise from that decision.

The primary source, Paul-Labrador, evaluated in the review looks very strong. It was conducted at the Cedars-Sinai Medical Center, one of the top ten medical research centers in the US, and is published in a journal of the AMA. It's included in many research reviews.

Also the review itself, despite being published in an unestablished journal, has some strong points. It is systematic, and was funded by NIH. It's peer reviewed, and the authors, Anderson and Taylor, are well credentialed. It comes out of the University of Virginia School of Medicine.

Is there any guidance on what weight these factors should have? Spicemix (talk) 13:43, 3 February 2012 (UTC)

There is strong consensus, developed over multiple threads on talk pages and noticeboards, including RSN, MEDRS, Project Medicine, and Fringe Theories, over many years, and involving many involved and uninvolved editors, that primary sources should not be used at all for medical claims in the TM articles. Fladrif (talk) 19:18, 3 February 2012 (UTC)

Psychology

Based on comments at Talk:Dissociative identity disorder today, it appears that we may need to specifically mention "psychology" as a topic area that is covered by this guideline. It's being asserted that since talk therapy involves no medications, then MEDRS doesn't apply.

I don't want to deal with this now, but perhaps in a few weeks someone will remind me about this. I think we'll want to solicit opinions from the folks at WP:PSY when deciding whether and how to address this perceived loophole. WhatamIdoing (talk) 04:23, 2 February 2012 (UTC)

WP:DUE

Should we add some text about WP:DUE when we address primary sources? Currently we have issues like this [25]

Reliable primary sources may occasionally be used with care as an adjunct to the secondary literature, but there remains potential for misuse. For that reason, edits that rely on primary sources should only describe the conclusions of the source, and should describe these findings clearly so the edit can be checked by editors with no specialist knowledge. In particular, this description should follow closely to the interpretation of the data given by the authors or by other reliable secondary sources. Primary sources should not be cited in support of a conclusion that is not clearly made by the authors or by reliable secondary sources, as defined above (see: Wikipedia:No original research). WP:DUE must also be keep in mind as many comments made may be of undue weight when added to a general medical article.

Comments?Doc James (talk · contribs · email) 07:09, 20 January 2012 (UTC)

Comment. Whatever else, I'd say that many comments made may be of undue weight is not clear enough to be really helpful here. - Smerdis of Tlön - killing the human spirit since 2003! 05:22, 22 January 2012 (UTC)

How about "nearly all comments made" --Doc James (talk · contribs · email) 18:26, 28 January 2012 (UTC)

We do need some sort of clarification of the UNDUE issue with respect to primary sources-- you can find a study to say just about anything, and adding such to articles gives undue weight to obscure topics. On matters that are well covered in secondary reviews, if something is left out, there's usually a reason. I hope we can find some wording to address this problem. SandyGeorgia (Talk) 19:05, 28 January 2012 (UTC)

If I might make a suggestion, how about something like: "When citing primary sources, particular care must be taken to adhere to Wikipedia's undue weight policy. Secondary sources should be used to determine due weight." Jakew (talk) 19:30, 28 January 2012 (UTC)

Sure that sounds good. Doc James (talk · contribs · email) 19:57, 28 January 2012 (UTC)

That sounds good to me, so I have added it. WhatamIdoing (talk) 20:09, 28 January 2012 (UTC)

I had a query about this particular change: what about tertiary sources? These often make a direct effort to provide a fair and balanced approach to a discipline whereas secondary sources make no effort to provide fair weighting when considered amonst others!

The last comment here [26] seems wise. Mindjuicer (talk) 00:08, 18 February 2012 (UTC)

The "last comment" says "See what the reference texts and textbooks say about the two views. Balance them accordingly". For most of MEDRS's purposes, reference texts and textbooks are secondary sources. WhatamIdoing (talk) 19:24, 20 February 2012 (UTC)

Sorry, missed your reply until now.

What about other tertiary sources, such as guidelines from national health boards and the WHO?

My concern is that WP guidelines such as this are used as weapons on many pages. Prominence of secondary sources is easily contested and can be skew. What is the reason for excluding tertiary sources which, by definition, should provide a balanced overview? --Mindjuicer (talk) 18:42, 23 February 2012 (UTC)

This guideline clearly identifies position statements/guidelines from recognized expert bodies as secondary sources, not tertiary sources. MastCell ^Talk 19:17, 23 February 2012 (UTC)

Am I missing something? Medical and scientific organizations says nothing about secondary sources. Nor does WP:MEDASSESS.

Also, I think it's clear that secondary sources can be tertiary sources as well. Or more accurately, the same publication may contain primary, secondary and tertiary statements. --Mindjuicer (talk) 20:04, 23 February 2012 (UTC)

Guidelines produced by national bodies that assess individual studies are by definition secondary sources. Your statement above declared them to be tertiary sources, which is not necessarily correct. Yobol (talk) 20:07, 23 February 2012 (UTC)

Look in the "Definitions" section (it's the first one). Look under "Secondary sources". You'll find: Examples include literature reviews or systematic reviews found in medical journals, specialist academic or professional books, and medical guidelines or position statements published by major health organizations. Emphasis mine. MastCell ^Talk 22:31, 23 February 2012 (UTC)

@Yobo1, agreed. The problem here may be the widespread assumption that if a publication appears to be more like a tertiary source then it cannot be a secondary source.

But are we not assuming that a) all secondary source publications are inherently more reliable than all tertiary source publications in spite of the latter's considered opinion on weighting and b) secondary sources will be reliably interpreted for due weight?

@MastCell, d'oh. Thanks. --Mindjuicer (talk) 23:18, 23 February 2012 (UTC)

"Secondary" is not an alternative spelling for "good". Neither is "tertiary". We've seen plenty of lousy secondary sources—and plenty of lousy tertiary sources, and plenty of lousy primary sources.

We fundamentally have no choice but to assume that editors will (attempt to) assign due weight to all of their sources. There is no viable alternative. Figuring out what's DUE is difficult, but there's nothing special about a secondary source that makes it harder to identify DUE weight compared to a tertiary source. WhatamIdoing (talk) 01:45, 24 February 2012 (UTC)

Let's take a NHS review compared to a meta-analysis. The review has due weight built in -- all you have to do is mirror it (along with other reliable 'overview' sources). However, the meta-analysis would merely draw conclusions on a small aspect of the treatment modality. There's no indication of due weight at all without first comparing it to all other secondary sources, with the importance of each potentially raising a separate argument.

Some editors will argue over anything they can, just to break momentum. So in the same way that MEDRS indicates a strong DUE preference for secondary sources over primary sources, should it not also indicate that "Secondary sources should be used to determine due weight, especially those that provide an overview of the subject"? I'm looking for language that doesn't raise a weak overview source over strong non-overview sources. --Mindjuicer (talk) 03:21, 25 February 2012 (UTC)

Meta-analysis

The claim that meta-analyses of independent RCTs provide the best evidence needs a citation (to a high-quality most reliable source).

Thanks! Kiefer.Wolfowitz 12:36, 5 February 2012 (UTC)

[27], considered by some to be the "bible" of EBM, states meta-analyses of RCTs to be Level 1a evidence, or highest level. Yobol (talk) 03:00, 7 February 2012 (UTC)

Additionally, per Wikipedia:Policies and guidelines#Not_part_of_the_encyclopedia, no such citation is needed. There are no reliable sources on the subject of what the Wikipedia community believes is the best evidence for supporting such claims in a Wikipedia article. WhatamIdoing (talk) 04:57, 7 February 2012 (UTC)

@Yobol,

Thanks for the clarification and citation.

FYI, "meta-analysis" graces the (long) list of (non–Harry-Potter) philosopher's stones in David A. Freedman's Statistical models. I've not read Ingram Olkin making such claims..., so the citation is very helpful.

@WhatamIdoing?

Please read "helpful" or "desirable" for my flawed "necessary".

The guidelines already cite 10 or so references, which provide further reading or substantiate a point that might be questioned by a sophisticated editor. Yobol's reference is similarly useful to the guidelines.

Thanks to you both! Cheers, Kiefer.Wolfowitz 06:23, 7 February 2012 (UTC)

I updated the article using this source. Page 99 states that systematic reviews of non-randomized studies are worse than meta-analyses of RCTs. Kiefer.Wolfowitz 06:39, 7 February 2012 (UTC)

Research reviews

MEDRS strongly recommends the use of research reviews, but gives little guidance as to how to determine whether a particular research review is reliable. Currently the only criteria are peer-reviewed secondary sources and publications on Brandon/Hill. Would it be useful to identify other considerations? For example impact factor seems already to be widely referred to. Listing on major indexes has recently been suggested at RSN[28], as have citation of the review and citation of other work by the authors. Other possibilities, raised here[29], are the standing of the authors of the review (well credentialed, widely published?) and the status of their research center; also a consideration of the quality of the research included in the review (publication of the primary sources in top medical journals? the primary research conducted at top medical centers?). Support from the NIH or other respected organization is another suggestion.

A further consideration could be whether the journal carrying the review is put out by a prominent association, organization, or university.

Would the inclusion of factors such as these be helpful? Spicemix (talk) 13:20, 7 February 2012 (UTC)

I'm not sure that the benefits would outweigh the costs. We have a fairly serious problem with POV pushers who want to discredit apparently good sources, and this would pretty much turn into a checklist of reasons to reject any publication that I personally disagree with. That's why the last paragraph of WP:MEDASSESS tells editors not to personally evaluate studies.

Impact factor and citations, in particular, are poor markers for quality. They tell you something about the publication's popularity or prominence. A review that gets cited ten times as an example of flawed thinking has the same citation count as a review that gets cited ten times for its brilliance. WhatamIdoing (talk) 19:30, 20 February 2012 (UTC)

Many thanks for this helpful response. Spicemix (talk) 11:24, 25 February 2012 (UTC)

Status of this page

Following up from a conversation elsewhere, I'd like to begin a discussion here on making several changes to this guideline:

Upgrade it to a policy page
Emphasize the importance of accuracy, the risk of harm, and the need for the strictest application of WP:V (in the same vein as WP:BLP)
Provide guidance for enforcement (perhaps including the creation of an actual noticeboard, a la BLP/N)

I think at least #1 should be pretty straightforward (the guideline has broad support, relative stability over the long-term, and serves the purpose of codifying in more detail the general policies on verifiability and reliable sourcing). The others will take some work and some back and forth. I'd suggest #1 be completed first, and then work can commence on improving the new policy to better describe the importance of MEDRS and support those working to enforce it. Any objections to posting the proposed policy template on the page as a first step? Nathan^T 19:49, 26 February 2012 (UTC)

Something else to consider. There's more to ensuring appropriate coverage of health and medicine articles than identifying reliable sources, and the guideline as it is goes beyond its current title. Might WP:Health and medicine articles be a more appropriate title for the long-term goal of focusing more resources in this topic area? Nathan^T 20:04, 26 February 2012 (UTC)

Nathan, you've basically got two different proposals here, and that will confuse the results below. How about discussing this some more and then starting a formal RFC? -- Brangifer (talk) 17:10, 27 February 2012 (UTC)

I support these measures; the prominence of Wikipedia as a trusted source of health information places a significant burden of responsibility on our medical article editors to ensure the greatest accuracy. I think a more stringent, careful attitude to content in this area, and less tolerance for borderline editing is called for relative to the rest of the encyclopaedia, and Nathan's suggestions represent steps in the right direction in this respect. Skomorokh 14:31, 27 February 2012 (UTC)

Support per previous. Kiefer.Wolfowitz 14:38, 27 February 2012 (UTC)

Support. I thought it was policy until recently, to tell the truth! It certainly should be. I'd also support the rename — as Nathan points out, the scope is broader than identification of reliable sources. Jakew (talk) 16:33, 27 February 2012 (UTC)

I've been beating this drum for a couple of years now, so I'm excited at the possibility that there might actually be support for something like this. I'm actually much less concerned about whether WP:MEDRS is a guideline vs. a policy. I want to know whether the community is ready to take this issue seriously. If so, then it doesn't really matter whether this is policy, a guideline, or even an essay; good things will happen. On the other hand, official "policy" status would probably head off at least one level of wikilawyering.
Not to look a gift horse in the mouth, but the cynic in me has some reservations. We've built something here that works. WP:MEDRS (in its guideline form) has been accepted by the community and even formed the basis of ArbCom findings. We have a group of knowledgeable, sensible editors who (while we often disagree on individual issues) share a commitment to high-quality medical information and thus "enforce" the guideline in a de facto sense. There is obviously still a vast quantity of wrong, misleading, or outright dangerous crap in this site's medical articles, but we have the tools right now to fix any of them that we turn our attention to (and we have fixed quite a few already).
More visibility is good, but along with a debate about policy status we're going to get the downsides of "community" involvement—namely, attention from the growing class of Wikipedians who lack or have lost interest in actual content and instead crowd onto whatever project-space discussion seems the most politically active. This guideline works because it was formulated, revised, and implemented by people with in-the-trenches experience. If it becomes a policy, then I'm worried it's going to become like most of our other policies—ossified, impossible to change without three preceding RfCs, and divorced from the reality of actual editing.
I realize that this is a cynical, if not frankly elitist, viewpoint, but after six years on this site, that's where I am. MastCell ^Talk 17:24, 27 February 2012 (UTC)

(ec) ::That's a fair point. (And the part about "editors" who don't add much content certainly applies to me -- at some point a couple of years ago, the feeling of negativity I got from reading and interacting in the metapedia overwhelmed my desire and ability to contribute regularly). I agree that the WP:MED / WT:MEDRS crowd of dedicated regulars should drive any changes to MEDRS and lead any effort to make enforcement more effective. It's also clearly true that drawing in a wider audience of Wikipedians will net both experienced and serious editors and people whose contributions will be less appreciated.

Drawing the parallels with BLP... For a long time it was the preserve of a small number of administrators and editors who worked tirelessly, and without much support or recognition. Over time, a number of factors contributed to repeated crises and controversies - administrator and editor burnout, decisive but controversial decision-making, attention from the media and external critics, and the involvement of high-profile internal actors (like Jimbo and the WMF board). While the ultimate outcome clearly benefited BLPs and their subjects, to say the process was painful for many community members is an understatement.

My hope is that the final product of BLP, and its status in the community, can be attained for MEDRS with more peaceful means. It may be that the best time to push for broader awareness of MEDRS and related issues is when the guideline, and its use, is at a sort of stable equilibrium and enjoys broad (if quiet) support. It may also be that the potential benefits aren't worth the chance at upsetting a system that, while far from perfect, "works" and represents years of heroic effort. It's for the participants in this discussion to decide if the uncertainty and risk to the status quo is worth the attempt to achieve a more complete solution. My personal opinion is that it's been unfair of the project to expect this Sisyphean task of monitoring and improving to be handled by a small group of people, and I think that you have suffered far more than is necessary to accomplish something everyone should agree is important. At some point we need to find a better way, a way that doesn't burn out great Wikipedians one after another year after year. Nathan^T 18:35, 27 February 2012 (UTC)

Goodness, where have you been all my life !?!?! Thanks, good stuff. But we've got work to do. SandyGeorgia (Talk) 18:39, 27 February 2012 (UTC)

All very accurate, we need to take this slow, and be very aware of how any proposal is formulated. An excellent interim step would be some sort of education of admins about how to determine reliable medical sourcing. I'm waiting for Colin to weigh in here, as I have some of the same concerns as MastCell, and the page isn't ready to be simply converted to policy. We need a careful discussion of how to give MEDRS more teeth, to make it work more like BLP vios, and wider visibility, regardless if it's policy. SandyGeorgia (Talk) 17:51, 27 February 2012 (UTC)

Somewhat related to these concerns, I'm a bit worried about the tendency for some editors to demand the best sources as an end, rather than as a means. No matter what the subject is, the source needs to be strong enough to support the claim being made. It does not need to be any stronger than that. We don't need a review article to support a lightweight claim like "Cancer is a disease". A very basic, non-medical dictionary is going to be good enough for that.
What we ultimately want is accurate, up-to-date information in the article. Using reviews and textbooks and other top-quality sources is nothing more than a means for getting accurate, up-to-date information into the article. If you can get accurate, up-to-date information into the article while using a relatively weak (but still adequately reliable) source, then that's actually okay: perfectly good material shouldn't be removed merely because it's followed by a merely adequate source instead of a stellar one—or, indeed, by no source at all. It must be possible to supply a reliable source for all material in articles; the source need not actually be named at the time of the original edit. WhatamIdoing (talk) 18:18, 27 February 2012 (UTC)

Yep. I regret that someone launched an RFC here before we had a chance to hammer out how the page would have to be formulated to give it teeth akin to BLP, addressing concerns such as those raised by WhatamIdoing. For example, the BLP page clarifies when certain types of sources can be used in BLPs. Sometimes we can use a primary source for basic info; sometimes not. SandyGeorgia (Talk) 18:21, 27 February 2012 (UTC)

And there we go ... a premature RFC before those knowledgeable have weighed in, and we even have an RFC proposal. Surefire way to crater anything useful. [30] Thanks. Do folks ever consider that before launching an RFC, you should have formulated what the RFC is about? And why do folks start "voting" before there's even something to "vote" on? SandyGeorgia (Talk) 18:24, 27 February 2012 (UTC)

Thank you ASCII, much appreciated. Now we can hammer something out more carefully, and then hopefully launch an RFC. Colin is out for the day, not feeling well, so I hope we can all put our heads together and come up with something after he's back on board. SandyGeorgia (Talk) 18:29, 27 February 2012 (UTC)

Yeah, the RfC tag is definitely premature. This needs thoughtful input from the experts in this area before it gets put to the wider community. Nathan^T 18:38, 27 February 2012 (UTC)

(groan) yes I think this page needs to become policy, but we must ensure all permutations and exceptions are covered. I really need to read it through when I have time (hopefully later today) and think it through. Casliber (talk · contribs) 19:30, 27 February 2012 (UTC)

One thing that strikes me is the different tones taken by the two documents (MEDRS and BLP). MEDRS reads like advice (appropriate for a guideline), while BLP is far more stern. As just an example of the tone difference, I've reproduced the intro to both below. There are benefits and drawbacks to both approaches, but it seems like MEDRS could be worded more forcefully and that this might provide better cover and support for people trying to enforce it. That's possible even without changing MEDRS to a policy, I think. Nathan^T 19:32, 27 February 2012 (UTC)

Yes. After consensus has been approached here, we should invite SlimVirgin and other BLP champions for advice. Kiefer.Wolfowitz 19:45, 27 February 2012 (UTC)

That will be interesting: Wikipedia:Featured article candidates/Female genital mutilation/archive1 revealed several instances of non-compliance with MEDRS, which resulted in POV. SandyGeorgia (Talk) 20:02, 27 February 2012 (UTC)

(ec) As I recall, BLP was an act-of-Jimbo, hence it was possible for it to be strongly worded from the beginning. Most other policies and guidelines lack that particular luxury, and have to adapt over time, with a gradual cycle of editors being slightly more strict than required by policy, then policy being updated to reflect that, and so on until a steady state emerges. I suspect that there would be more resistance to making MEDRS policy now if it were strongly worded; it would be easier, I suspect, to do that on an incremental basis once it had become accepted policy.

I'm a little concerned by the above remarks that we might be "letting the perfect be the enemy of the good" (to paraphrase Voltaire). MEDRS doesn't have to be perfect before it becomes policy, and it doesn't have to cover every conceivable scenario. It just needs to enjoy strong support in the situations it does cover. MEDRS is a fantastic document: it helps tremendously in implementing and clarifying many other policies in the context of medical articles, adding extremely useful guidance of its own. It deserves to be policy, and that shouldn't in any way stop it from improving further. Jakew (talk) 19:46, 27 February 2012 (UTC)

Yes, but almost all of us who have to apply this guideline in our every day editing still acknowledge that we have some niggles to work out (along the lines of WhatamIdoing's post) before it's ready to be upgraded to a place where it's useful to shoot badstuff on sight as we can with BLP. She gives the example of a dictionary definition of a condition that wouldn't necessarily require sourcing to a medical journal secondary review. That is sometimes true. And it is sometimes not true (the Merck Manual for years had an incorrect definition of Tourette syndrome, and they can still be found many places, so we get into territory of when better quality sources disagree). It gets even better: the New England Journal of Medicine (a most prestigious journal) once printed a faulty definition of Tourette syndrome (damagingly so, claiming that everyone with TS also has ADHD, OCD and disruptive behaviors, utterly untrue), later addressed via a letter to the editor, but they never retracted the article (the faulty part was only one Venn diagram). We've got to nail down the spectrum of possibilities so that we end up with something that allows us to shoot poorly sourced info on site, while not demanding too much of sources. There are also situations where it is OK to use a primary source: I was obliged, unwillingly, to work on klazomania after a group of students adopted it. There are no reviews on that topic, so it was OK to judiciously use primary sources to describe symptoms. I can think of lots of work we need to do here to cover all bases, but I believe if we do that, we can have it ready for a discussion of promotion to policy. We need to end up with something where we know when we can strongly enforce the removal of inaccurate medical text when there are secondary reviews that give better peer-reviewed info. SandyGeorgia (Talk) 19:59, 27 February 2012 (UTC)

Okay, those are fair points. But please remember that an awful lot of good can come of it simply becoming policy, so let's get the right balance between getting it just right and making it policy soon. Jakew (talk) 21:57, 27 February 2012 (UTC)

While I note Klazomania isn't in "Category:Rare diseases", would no results from a PubMed search "review[pt] Klazomania" or Category membership be a reasonable criterion for allowing primary sources? Something like the DOIBot could walk the Medical articles to flag citations with a PMID that don't have "publication type" review (probably), if useful. RDBrown (talk) 02:17, 28 February 2012 (UTC)

Support after helping out here I fully support MEDRS and I think It would be better for it to have some teeth --Guerillero | My Talk 21:11, 27 February 2012 (UTC)

Support appreciated, but let's not !vote on anything, please, until some issues are hammered out. We're not ready for policy prime-time yet :) SandyGeorgia (Talk) 21:27, 27 February 2012 (UTC)

Focus of goals

User:ImperfectlyInformed changed the above thread title from "Focus of goals" to "Preliminary discussion about promoting to policy" for transparency.

And I changed it back, since your new title is even more misleading. No regular editor here believes we are ready to promote to policy. The discussion is about our goals. If we get to the point of having something ready to be policy, then the title can suggest that. SandyGeorgia (Talk) 16:03, 28 February 2012 (UTC)

Hi. Not up to big thoughts today :-(. I'm very encouraged to see folk wanting to get serious about medical topics in the same way as BLP. I share some of the concerns raised. I'm not totally sure than "upgrading" MEDRS to policy is what we're after. Perhaps we should have a discussion about what we really are after, since WhatAmIDoing is quite right that sourcing is a means to an end -- and also that there is a hierarchy of sourcing quality that needs to be matched with the claim being made (extraordinary claims / extraordinary evidence). The gold standard randomised controlled trial, or meta analysis of such, tends to only address efficacy questions, which is really a small part of health related information. When dealing with a dispute I've often found citing WP:WEIGHT to be more helpful than MEDRS. Because weight is a reason to need secondary sources. I also find the "reliable" aspect to cause confusion sometimes. People naturally don't understand why we might say a paper published in a respected peer-reviewed journal might be "unreliable". WhatAmIDoing frequently says that "primary" is not another word for "unreliable" and that is true. A primary research paper can be an extremely reliable document about the research (except if Wakefield wrote it). But we generally shouldn't be writing directly about research but rather about the knowledge that research brought about. And the clinical application and generalisation of that research requires a secondary source. That isn't an issue of "reliability" but of NOR and WEIGHT.

Oh, this is a bit of a wandering brain dump. Sorry. So I think we should discuss the nutshell of our goals and principles. The issue of scope is also important. BLP has a fairly specific scope that is easy to address. The issue of power is also important. BLP gives editors absolute rights to remove text on sight. Are we after that and if so, which sorts of medical facts would be subject to such? What would be the consequence of that on our articles, which may not be well sourced at present. Would someone take the new policy and start gutting medical articles of information that actually was ok just because it wasn't sourced to a review in the Lancet? Colin°^Talk 21:33, 27 February 2012 (UTC)

By whatever process we get there, I believe our goals should be:

Better education throughout Wikipedia about what a MEDRS-compliant source is, how to find reviews, when they should be used, why not all medical journal reviews are the best sources (per Colin above), etc-- general education. I am often shocked to realize that MEDRS isn't well known at GAN, DYK, Articles for creation, New page patrol-- so many places that could be the first place to educate offenders. Many editors see a journal citation, or a New York Times citation, or a book written by a lay advocate, and think the article is well sourced.
A page that will better allow us to shoot the really bad non-compliant poorly sourced text on site, and not have to spend weeks arguing over it.

SandyGeorgia (Talk) 21:57, 27 February 2012 (UTC)

One benefit to "upgrading" it is the follow-on benefits in other fora - policy status and prescriptive language avoids the "it's just a guideline" or "it only says you should try to do X, not that you have to" objections. The question on the consequence to articles that Colin raises is a good one. "Harm" in the BLP sense is comparatively easy for a layperson to identify, and scaling the severity of harm is somewhat straightforward. How would someone not practiced in applying MEDRS calibrate how aggressive to be in addressing questionable claims? Judging the potential for harm in the MEDRS umbra of articles is likely to be difficult for a non-expert. Nathan^T 22:42, 27 February 2012 (UTC)

Related to the point about "it's just a guideline", you might like to read WP:The difference between policies, guidelines, and essays if you aren't familiar with it. WhatamIdoing (talk) 04:35, 29 February 2012 (UTC)

I am indeed. I would say that just like there is a vast gulf between how we are supposed to see administrators and how they are seen in practice, there's a real difference between how policies, guidelines and essays are supposed to interact and how they are actually used day to day. Policies and guidelines are both supposed to represent the Wikipedia consensus, and both can be enforced, and policies are descriptive not prescriptive, etc. etc. but in practice, a policy is a much bigger club than either a guideline or essay. Nathan^T 15:10, 29 February 2012 (UTC)

Background and opposing view

Oppose Is this a serious proposal? If so, the title of this thread should be changed or a subthread should be added that clearly states the proposal, e.g. "Proposal to upgrade MEDRS to a policy". Someone should also probably drop a note in the village pump and probably at some point at the top of the watchlist. I had to fix the RfC ta.

I think this policy already gets abused enough as a guideline. That's not to say I don't think it should be a guideline, but it really gets abused. For example, I've had a perfectly good secondary source published by the American Diabetes Association blocked (Talk:Statin/Archive 1#Doc News, MEDRS, and doctors' dismissal of complaints); in the archives, the most recent (Wikipedia talk:Identifying reliable sources (medicine)/Archive 5) shows comments about it being used to remove research sections (Wikipedia talk:Identifying reliable sources (medicine)/Archive 5#MEDRS and research sections) and about editors overlooking the finer details of the guideline and its neglect of research article value(Wikipedia talk:Identifying reliable sources (medicine)/Archive 5#The value of peer-reviewed studies reflecting developments since the last review is greatly overlooked.). There's a lot of need for editorial discretion in editing in order to make a balance, highly-valuable. MEDRS tends to be interpeted as "reviews are required". However, reviews are a very small fraction of the research out there. In many lines of research, they don't exist. In many others, there's none that are good or up-to-date. Back in Wikipedia talk:Identifying reliable sources (medicine)/Archive 1, I was involved in some spirited (probably much too spirited) debate. As I recall it User:Eubulides drafted much of the original language. In Archive 2 based on Wikipedia talk:Identifying reliable sources (medicine)/Archive 2#Should we make this a guideline.3F this was promoted to a guideline, although it was opposed by 3 of the 17 or 18 voters (User:Paul gene, User:Una Smith, and User:Mihai cartoaje; I abstained). The promotion over 3 led to a controversy when one of the opposers, User:Paul gene (a self-professed pharmaceutical researcher), said that it had been raised to a guideline inappropriately (Wikipedia talk:Identifying reliable sources (medicine)/Archive 2#Rfc Promotion of MEDRS to guideline). Eventually, when User:Paul gene stopped protesting, everything went on as usual, but there's frequent discussion and concern about the ability to use MEDRS to keep out perfectly encyclopedic information, and to make Wikipedia more susceptible to biases such as the pharmaceutical lobby. II | (t - c) 07:40, 28 February 2012 (UTC)

II, the RFC was agreed to be a mistake and was retracted. You didn't "fix" it, you've just buggered things up. We absolutely don't need a pile-on of support/oppose comments at this stage. We are at the very early stages of working out what should be proposed to the community. That might be a change to MEDRS status but it might be a new policy page.

The above linked discussions don't really help your case, I'm afraid. For example, Wikipedia talk:Identifying reliable sources (medicine)/Archive 5#MEDRS and research_sections contains a friendly discussion where everyone agreed that the information could and should come from secondary sources in order to determine the appropriate weight to give to the new research. I agree that some editors may be over-zealous in their application of MEDRS but that is an editor-behaviour issue, not a content-guideline issue. For the vast majority of situations, MEDRS is appropriately used to inform editors of the standard of sourcing we require for medical claims. I do agree with you that there is more useful information than that which appears in reviews.

Paul gene was a classic example of an academic editor who didn't get Wikipedia. He believed the encyclopaedia should be written by expert editors drawing from the primary literature. He thought he could do a better job than the writers of reviews in medical journals, and should be allowed to do so on Wikipedia.

Wikipedia's guidelines are designed to keep out biases, whether from the pharmaceutical lobby or the alternative medicine lobby or the megavitamin lobby or whoever. Our secondary sources are imperfect and some may indeed be biased, but they are all we've got. Individual editors may think they are unbiased and a better arbiter of the truth than the published literature, but the evidence shows that many of the editors on Wikipedia come with their own bias and will cherry-pick the primary literature in order to push that bias on WP. Our system of deferring to the secondary literature is not perfect but it works reasonably well given our editor-base. The alternative is for Jimbo to pay experts he trusts to write articles for him. I believe he tried that and it didn't work. Colin°^Talk

Oppose I too oppose for now. Here are my concerns:

Scope: What articles is MEDRS applicable to?
- Is it applicable to any medicine-related article?
- Is it applicable to the whole article?
- Should the same stringent guidelines apply to sections on biochemistry/genetics, veterinary medicine and pharmacology? Have people from these fields been asked for their opinion?
Too stringent on research sections.
- Reviews are of course preferable, but stripping out primary research results without replacing with a review is, in my opinion, usually unhelpful.
- Results in rats/model organisms should not be generalised to humans. No debate. But that doesn't mean the results are not useful or significant to non-human medicine. Yet they are unwelcome in research sections.

I feel MEDRS pushes to make Wikipedia into a human medicine textbook. By all means be stringent on human medicine material, but it also needs to be more inclusive. pgr94 (talk) 11:00, 28 February 2012 (UTC)

Pgr94, it would be really helpful if you rephrased your comments to remove the !vote. We really are not at the stage where something concrete is proposed. MEDRS does specify its scope, so I suggest you read it. There's a debate to be had about how existing text that is poorly sourced should be handled, but we need to shift the culture of editing to where editors are not adding text sourced to primary research papers nearly as often as they are. Do you agree that that text+source combination shouldn't have been added in the first place? The person who decides whether rat experimental results are of interest to humans is the expert writing a published secondary source. They are only "unwelcome in research sections" where editors have added them based on their opinion of their importantance and relevance to humans, and have failed to satisfy WP:WEIGHT by indicating which secondary sources support that view. I don't really understand your last sentence. Colin°^Talk 11:12, 28 February 2012 (UTC)

It looks like we have two participants here who didn't read the page (that's helpful), and decided to add unconstructive commentary; we know the RFC was premature, it was inititated in good faith, based on a post that wasn't a clear proposal, and that's why it was removed. If folks weighing in on discussions actually read them, it always help. I've cleaned up more of the formatting mess introduced by II, the link to Eubulides' correct name, and no-- Eubulides was not instrumental in this page, and even if he were, discuss the content, not the contributor (he was quite an excellent editor, but whatever). SandyGeorgia (Talk) 14:25, 28 February 2012 (UTC)

Eubulides wasn't involved in the initial draft, but did become a significant contributor to the text. See this contributor list. I regard that as a very good thing. Colin°^Talk 14:50, 28 February 2012 (UTC)

SandyGeorgia, I believe my comments are constructive; perhaps they just don't match your point of view. I am not in favour of crystallizing the current MEDRS page into policy at present (which is my understanding of the proposal in the opening paragraphs of this discussion). pgr94 (talk) 14:39, 28 February 2012 (UTC)

Ok. Let's move on from that. Can you deal with my response above. I don't think your "oppose" reasons are valid. We agree that it is more helpful to rewrite text with better sources, but ultimately, editors should have the right to remove contentious medical facts that aren't reliably sourced -- and policy already allows them to do that (see below). I've addressed the scope/rats comments above. Could you respond? Colin°^Talk 14:45, 28 February 2012 (UTC)

I noticed Colin above asked Pgr94 (talk · contribs) to remove the oppose vote. Why isn't the same question being asked of everyone who has bolded support statements at the top? In any case, I don't think I'll ever sign on to promote this to policy. Guideline is just a better semantic fit. Also, I did indeed read through the thread, but there was no notice at the top whatsoever that the proposal was withdrawn by its initiator. Just because a few people think a proposal is premature doesn't mean it gets withdrawn. If reasonable processes are followed to make threads clear and transparent, then there won't be confusion. The title of this thread, for example, is clearly not a reasonable title. II | (t - c) 15:55, 28 February 2012 (UTC)
- Because I couldn't be arsed. There's no conspiracy, II. I agree with you that as it stands, this guideline is a guideline and isn't really worded appropriately as policy. Everybody thought the RfC was premature and is best withdrawn until we've collected our thoughts. It could be quite some time before we work out what to propose to the community. It is difficult to change titles because people link to them and anyway that was the initial question, which has since matured. — Preceding unsigned comment added by Colin (talk • contribs) 16:05, February 28, 2012
- II, please read more carefully; your summary is incorrect. People voting support were asked several times not to, and the discussion of the removed RFC is there. It should also have been very clear that I was unpleased that an RFC was launched by someone new to this page well before anyone who regularly participates here had said we had anything close to policy. Please avoid sidetracking the discussion of where we want to head with this page, if anyone tries to make it policy as it stands, you will find every regular medical editor here most likely opposing, the page is by no means defined by anyone yet as being close to ready for promotion to policy, thank you for not being disruptive.SandyGeorgia (Talk) 16:08, 28 February 2012 (UTC)

II, I initiated the discussion on this page, but not the RfC. It wasn't intended to be an RfC, or to go "primetime" beyond the people on this page. Nathan^T 15:07, 29 February 2012 (UTC)

Psychology and other health-care disciplines

The policy should cover "human health-care" generally: psychology; nursing, scientific and therapeutic touchy-feely; dentistry, etc. Kiefer.Wolfowitz 19:58, 27 February 2012 (UTC)

While that would almost certainly be a good result, the process of getting some of those adopted might lead advocates of fringe practices to undermine any progress towards such an end. On the premise that perfection is the enemy of the good, we should probably set them aside and focus on EBM. LeadSongDog come howl! 22:55, 27 February 2012 (UTC)

I agree. If it's EBM-supported, it should be covered. If it's not covered by EBM, it's a belief system--one that may well be true and functional, but not one that can support specific claims that will stand up methodologically to, say low-dose aspirin's utility vs. MI. Jclemens (talk) 07:24, 28 February 2012 (UTC)

Kiefer, I believe you mean "proposed" policy, and we've not yet got anything close to a policy proposal here. SandyGeorgia (Talk) 16:41, 28 February 2012 (UTC)

Generally, we call out specific healthcare-related disciplines as we discover a need for it, i.e., as some editor tries to claim that the failure of MEDRS to explicitly say that _____ is 'medicine-related' means that they have permission to use inappropriate sources. That happened repeatedly with alternative medicine (now named), and it recently happened with psychology (see section above). But I don't think that we've had any similar problems with nursing or dentistry, so I'd avoid adding them. WP:Nobody reads the directions anyway, and the longer they are, the even lower chance that anyone will read them. WhatamIdoing (talk) 04:43, 29 February 2012 (UTC)

Burden

The policy WP:V says:

The burden of evidence lies with the editor who adds or restores material. You may remove any material lacking an inline citation to a reliable source. Whether and how quickly removal should happen depends on the material and the overall state of the article; consider adding a citation needed tag as an interim step.¹ Editors might object if you remove material without giving them time to provide references. It has always been good practice to try to find and cite supporting sources yourself. Do not leave unsourced or poorly sourced material in an article if it might damage the reputation of living people; you should also be aware of how the BLP policy applies to groups.

¹: It may be that the article contains so few citations that it is impractical to add specific citation needed tags, in which case consider tagging a section with {{unreferencedsection}}, or the article with {{refimprove}} or {{unreferenced}}.

²: Wales, Jimmy. "Zero information is preferred to misleading or false information", WikiEN-l, May 16, 2006: "I can NOT emphasize this enough. There seems to be a terrible bias among some editors that some sort of random speculative "I heard it somewhere" pseudo information is to be tagged with a "needs a cite" tag. Wrong. It should be removed, aggressively, unless it can be sourced. This is true of all information, but it is particularly true of negative information about living persons."

it goes on to say

Content related to living people or medicine should be sourced especially carefully.

Now, I'm sure we all agree that in an ideal world, where experienced editors had limitless free time and limitless access to high quality sources, that any poorly sourced medical facts would be rewritten based on the best sources according to WP:MEDRS. That's what WP:PRESERVE encourages, though that policy also accepts there are times "when it might be more appropriate to remove information rather than to preserve it." WP:BURDEN has always placed the burden of reliable sourcing onto the editor who adds or restores the material. I disagree with WhatAmIDoing above when she says "the source need not actually be named at the time of the original edit". That's only true of material unlikely to be challenged, which is fairly rare for a health claim. We should not, of course, sanction an editor gratuitously removing text that wasn't using the best possible source. However, if an editor reasonably believes the text could be wrong or misleading, or the source fails to support the text, then they are justified to remove that text without supplying an alternative. Let's be clear that WP:V says "You may remove any material lacking an inline citation to a reliable source." All WP:BLP does, in effect, is to add some exclamation marks to that sentence, and removes the option of doing nothing. Here's what it says about sourcing:

Wikipedia's sourcing policy, Verifiability, says that all quotations and any material challenged or likely to be challenged must be attributed to a reliable, published source using an inline citation; material not meeting this standard may be removed. This policy extends that principle, adding that contentious material about living persons that is unsourced or poorly sourced should be removed immediately and without discussion. This applies whether the material is negative, positive, neutral, or just questionable, and whether it is in a biography or in some other article. Material should not be added to an article when the only sourcing is tabloid journalism. When material is both verifiable and notable, it will have appeared in more reliable sources.

A number of editors have suggested that medical facts, or health-related information, should be on the same level as facts about living people, particularly facts that may damage their reputation. So what sort of medical facts are we concerned about? How would we identify the sensitive ones from the everyday ones that can be treated like any other fact on WP? Should editor actions be similar to BLP? Will edits enforcing this policy have the same protection as BLP? --Colin°^Talk 13:17, 28 February 2012 (UTC)

I'm going to ignore your questions for now, since they will require more thought.

WP:V and WP:BLP require inline citations for exactly four types of information. But with the exception of contentious (not all) matter about BLPs, no policy puts a deadline on providing those citations. Adding non-BLP material without adding a citation is 100% 'legal' under the existing policies. You can't revert its removal if it's been removed, but WP:V does actually permit you to add the unsourced (but verifiABLE) material (once). WhatamIdoing (talk) 04:49, 29 February 2012 (UTC)

Hmm. WP:V says "material...must be attributed to a reliable, published source using an inline citation" and the responsibility for doing this is "the editor who adds or restores material". Where does it indicate that this action can be indefinitely postponed (as long as presumably you intend to do it)? Where does it say, then, that it cannot be postponed when one restores text? Both addition and restoration, to me, seem to have the same demands put upon them, and it is a demand that should be met with in a reasonable time frame (such as during an edit session). Are you saying that an editor who every day adds "liver disease" to a random drug article's adverse effects list, without supplying any source, would be editing perfectly "legally" and could not be stopped? They could simply claim that they will do it "some day". Colin°^Talk 09:28, 29 February 2012 (UTC)

Basically, yes: you can add unsourced (but not impossible-to-source) material. All you need to do is declare that it's not a direct quotation, that it's not contentious matter about a living person, and that you personally thought it was not information that was WP:LIKELY to be challenged. Assuming those three conditions, your unsourced addition is just fine unless and until it is specifically WP:CHALLENGED.

BURDEN only addresses one point: the identity of the person responsible for providing a citation. Specifically, BURDEN exists so that when I add the unsourced claim of "liver disease", and you challenge it (e.g., by removing it), then I can't tell you "Well, if you want a citation that badly, then go add one yourself."

(If you really want to make the person stop adding , you'll have to follow the general dispute resolution pathways, probably based on an allegation of disruption, not merely invoke WP:V.) WhatamIdoing (talk) 17:51, 29 February 2012 (UTC)

Please educate me if I'm wrong, but I don't believe one can do that on a BLP (or at least not continue to do it after one revert), and if I understand correctly, our aim here is to define under what circumstances similar would apply to medical statements. In other words, if someone adds an uncited birthdate to a BLP, and I have good reason to doubt it but can't cite the actual birthdate (let's say I know the person in real life, but bd is not citeable), if someone continues to add, I can cite BLP and not be subject to 3RR. SandyGeorgia (Talk) 18:15, 29 February 2012 (UTC)

Ok, to avoid talking past each other, I'm totally not interested in text that doesn't clearly require a citation. We're talking about text that says a drug has liver disease as a side effect or that eating some fruit cures cancer. So the person claiming they didn't think it was "likely to be challenged" is either lying or lacks a clue. Does policy make that action "legal". I'd say no. They've added some text to Wikipedia that breaks our policies. We all make mistakes but if they keep doing after being asked not to, and asked to fix things, then we'll get round to stopping them at some point.

The problem with the WP:V text is that it is a passive requirement of the text rather than an active requirement of the editor. And as you interpret it, even though BURDEN places a requirement on the text-adding-person, it doesn't explicitly say they should have done it in the first place. If the policy said "When adding or restoring material [that requires an inline citation] you must supply an inline citation" then we'd be all clear about where things stood. Why can't it be this clear when it uses clear language about who should or may remove the text. When my wife tells me the bins need emptied, she's not volunteering to do it herself, or expecting someone else to do it for us. Has this been discussed on WP:V before? Colin°^Talk 18:32, 29 February 2012 (UTC)

I suspect another place where we're getting into trouble here is WhatamI's concern over "five digits on a hand" getting a citation request, and I submit that is already covered at WP:V and we shouldn't get hung up on that territory. (Besides, even in that case, it looks to me like the cn tag was just applied in the wrong place-- would have been appropriate if moved a few words over, to cover the "other" conditions resulting in other than five digits.) SandyGeorgia (Talk) 18:36, 29 February 2012 (UTC)

No, I'm not worried about common knowledge here. The fact is that WP:V doesn't require the citation to be added, as Colin puts it, "in the first place", i.e., at the time of the original edit. Some editors believe it should, but it doesn't actually impose that requirement.

As for lying vs cluelessness, challenges are so rare on low-traffic pages that even the most extraordinary claims are not WP:LIKELY to be challenged, because ot's not even likely that anyone will read the page, much less tag it. LIKELY is effectively a lower standard than WP:When to cite. WhatamIdoing (talk) 19:11, 29 February 2012 (UTC)

I'd have thought "challenges" are pretty rare full stop. I mean WP is so full of unsourced crap that nobody reads that you could then statistically say WP:V isn't really required at all, and the 1% that matters to anyone is an edge-case. I've always found "challenged" problematic because it seems rather aggressive, like you're saying the editor is lying or a gullible fool if they believe that. Perhaps your essay should define "challenge" to mean "challenge the author to supply a source" rather than "challenge the author's integrity" or some other slur. I'm not happy that you say "likely to" includes the "likely to even be read" or "likely to care much" aspect. Surely a better assumption would be an editor, reading and interested in the page contents, is likely to want a source to confirm the veracity of that statement. But I suppose this discussion belongs on WP:V. I'm reluctant to go there just now because they are too busy fighting over the Truth, and because I've unwatched all the policy pages because they are a huge timesink. As for "in the first place", think we shall have to disagree on "it doesn't actually impose that requirement" as I think that is up to how someone reads the policy as currently worded (it should make it clear one way or the other if folk have agreed that much). --Colin°^Talk 19:39, 29 February 2012 (UTC)

You're right that WP:V is a mess at the moment. But if you'll remind me when (if?) that mess ever dies down, I'd be happy to propose a clarification about timing at BURDEN. Right now, it'll just get lost in the shuffle. WhatamIdoing (talk) 23:31, 1 March 2012 (UTC)

Examples

Adding in Colin's examples [31] to mine: SandyGeorgia (Talk) 15:39, 28 February 2012 (UTC)

I suggest we might focus the discussion better if we work from examples-- if we can put forward the range of possibilities, from least to most harmful, it might help others:

Poorly sourced information that is nonetheless correct (we need an example). Sometimes an experienced medical editor looks at an article, sees nothing incorrect, but doesn't have time to add the secondary review sources (or doesn't have journal access). What should be done in this case? Request citations, remove text to talk, etc?
See, for example, splenectomy. The indications section appears essentially correct but is unsourced. Likewise, later sections appear to correctly outline the need for vaccination and the important risk of overwhelming post-splenectomy infection, but are also unsourced. MastCell ^Talk 16:55, 28 February 2012 (UTC)

See this fact tag, added to the number of digits normally present on the human hand. We don't want information that is known (to the average child) to be correct to be challenged, much less to be cluttered up with a citation to a "secondary review source". You're simply not going to find a meta-analysis on the number of digits on the human hand. WhatamIdoing (talk) 04:52, 29 February 2012 (UTC)
I don't think we should get hung up on a case like that. Common knowledge is already covered by WP:V, and I suspect that person simply put the tag in the wrong place-- move it seven words to the right and it makes sense. SandyGeorgia (Talk) 18:38, 29 February 2012 (UTC)
Oh, I suspect the IP was having a laugh. And quite a good one really. But yes, can we just assume that this MEDRS discussion concerns something important, non-obvious, etc, otherwise it is going to be really tedious when WhatamIdoing (rightly) reminds us each time of all the exceptions to requiring a source :-) Colin°^Talk 19:11, 29 February 2012 (UTC)
[Added per Colin] Poorly sourced information that the editor has no idea whether correct or incorrect. For me, this is a common occurrence. This might be an inconsequential fact like the pills are blue but it might be a signficant fact like mentioning liver failure as a side-effect.
[Added per Colin] Poorly sourced information where the text is not fully supported by the source but may in fact be correct. An example would be an article saying that a drug is successfully used for treating a condition, and citing a drug trial. The drug trial doesn't establish clinical use, we'd need some clinical guidelines for that, or similar. What should the editor do?
Basic factual error, sourced to an outdated or contradicted review, or demonstrably wrong according to better or newer sources: an example of something like that would be if someone were to add, based on Jankovic's NEJM article, that the definition of Tourette syndrome includes ADHD, OCD, and disruptive behaviors. Basic factual error, only found in a Venn diagram in one journal article, contradicting hundreds of secondary reviews (and the DSM definition). Remove on sight as basic factual error that stigmatizes via mischaracterization persons with TS. Would BLP-type restrictions to reinstating something like this apply?
Basic factual error, poorly sourced (need an example). Remove on sight. Once removed, BLP-type provisions apply (don't reinstate, not subject to 3RR).
[Added per Colin] Text that violates WP:WEIGHT. Often text based only on primary research papers does this.
[Added per Colin] Text that violates WP:NOR. For example, stating or implying that rat research might be relevant to a human disease, citing the rate research primary research paper, rather than a review of the human disease.
Text that is not sourced according to MEDRS, but is not dangerous per se, although introduces POV or in one case, cultural stigma. The text from Female genital mutilation, stating a 10% death rate because one lay advocate claimed that in a book, with no citation or study or data given (based on a midwife's opinion), and not contained in any of the secondary reviews on the topic. There are, in that case, good secondary reviews available, but the article (at one point, not sure if it has since been fixed) ignored the medical reviews and the responsible and culturally sensitive tone they took to the issue in favor of lay advocate books against female circumcision. This is a case where inferior sourcing introduces POV. Do we shoot on sight, or just label the article as unbalanced?
Poorly sourced information that is dangerous: examples abound in PANDAS. There is an abundance of secondary journal reviews covering this unproven hypothesis, the problems with that hypothesis have led to "internet-armed parents" (terminology used in one secondary review discussing the PANDAS problem, whereby parent advocates are making faulty and dangerous choices based on misinfo spread on the internet) making poor medical decisions about the use of antibiotics, IVIG and other interventions that endanger the health of the child as well as society (over-use of antibiotics based on a controversial and unproven hypothesis). Advocates perennially chunk in primary-sourced text that is contradicted, unproven, unreplicated or otherwise unsupported by secondary reviews. Shoot on sight and apply something that needs to be developed akin to WP:BLP to dangerous poorly sourced information. The distinction I'm attempting between this (No. 9) and the previous (No. 8) is in the realm of misinformation that is dangerous to one's health (theoretically, no one will die because the incidence of death in female circumcision is overstated-- that's just a matter of POV).

I don't know if I've covered all bases, but we need to somehow define how a new guideline or policy page would be applied to different levels of severity. Better examples might be supplied. SandyGeorgia (Talk) 14:57, 28 February 2012 (UTC)

6 Poorly sourced information that the editor has no idea whether correct or incorrect. For me, this is a common occurrence. This might be an inconsequential fact like the pills are blue but it might be a signficant fact like mentioning liver failure as a side-effect.

7 Poorly sourced information where the text is not fully supported by the source but may in fact be correct. An example would be an article saying that a drug is successfully used for treating a condition, and citing a drug trial. The drug trial doesn't establish clinical use, we'd need some clinical guidelines for that, or similar. What should the editor do?

8 Text that violates WP:WEIGHT. Often text based only on primary research papers does this.

9 Text that violates WP:NOR. For example, stating or implying that rat research might be relevant to a human disease, citing the rate research primary research paper, rather than a review of the human disease. Colin°^Talk 15:11, 28 February 2012 (UTC)

Added to list above. SandyGeorgia (Talk) 15:44, 28 February 2012 (UTC)

Something like PMC 3227668 might be helpful to document why mass-media sources are unsuitable.LeadSongDog come howl! 04:34, 1 March 2012 (UTC)

On a vaguely related matter, I wonder if we could cite something like this to illustrate why advocacy sites on the web can be unreliable? Jakew (talk) 10:49, 1 March 2012 (UTC)

Policies, Guidelines, Standards, and Procedures

At one point, I wrote a large chunk of IT policies for a Fortune 100 company I'm sure you've all heard of. In that system, "policies" were broken down into: 1) Policies, a one page general statement of management's expectations, roughly analogous to our pillars, 2) Standards, specific statements of expectation on individual topics (e.g., no computer except an IT approved firewall shall be connected to both an internal and external network), and 3) Procedures, actionable means to accomplish those standards, where each procedure was hierarchically associated with a specific standard.

Maybe we need something like that here? Rather than trying to promote "identify reliable sources" to a policy on its own, we would be better served by a succinct policy that says something to the effect of "Medical articles or articles making specific medical claims must use sources according to (identifying reliable sources) to support those claims, or such claims must be reworded to make it clear that they are unsupported by scientific research". That way, we get a quick skeleton of a policy in place, which the community can all agree on in principle, and the guideline, the successor to this page as it stands, continues to evolve. I know that's not really the Wikipedia Way, which seems to be to shoehorn every last thing into a policy page and fight over changes, but maybe it would be worth trying in this case... Jclemens (talk) 15:42, 28 February 2012 (UTC)

I don't think there's any such thing as a "quick skeleton, which the community can all agree on in principle"; I can think of many editors who will object to your proposal. Slow and steady does the job, and it should be noted that no regular medical editor participating in these discussions believes that we yet have anything in shape here to be be promoted to policy-- there are issues to be resolved, and we should go about doing that slowly and systematically. The rush to RFC, or a jump to quick solutions, is detrimental to long-run solutions. SandyGeorgia (Talk) 15:50, 28 February 2012 (UTC)

I was quite surprised at how long WP:BLP was. I agree it would be helpful to start small but also agree with Sandy that there is no rush to establish something as polciy. MEDRS contains a lot of advice, as a guideline might, rather than just stating succinctly what our goals are and our main means of achieving those goals. A medical policy page could have a similar nutshell to BLP:

Material making medical or health claims must be written with the greatest care and attention to verifiability, neutrality, and avoiding original research

The same "we must get it right" and "written conservatively" attitudes would apply too. I do believe the focus on a policy page should be wider than just talking about "identifying reliable sources", which in some way is an unhelpful goal. The reliable/unreliable aspect is great for explaining why newspapers are poor sources, but not for why primary research papers in a peer reviewed journal are often "inappropriate sources" for the sort of text folk want to add. The issues we face involve all three of the above content policies, not just the appplication of WP:V to medical text. The choice and correct use of sources is a consequence of applying those policies rather than blindly driving the content. Colin°^Talk 15:59, 28 February 2012 (UTC)

By dint of comparison, I thought I'd take a look at the HONcode principles. Unless I'm misunderstanding, we should be able to meet all of them except principle 1 ("Authoritative: indicate the qualifications of the authors"), which would fall afoul of "anyone can edit" by requiring that editors out themselves. Still, it seems to me that something approaching that could help guide our policy development. We could in theory have volunteer identified reviewers put their real names on a specific revision of an article to say they found it was reasonably complete and current as of that (permalinked) revision. Such reviews wouldn't have to be either internal or external, each could be accomodated.LeadSongDog come howl! 17:05, 28 February 2012 (UTC)

HonCode is a massive failure (I know of cases where they have been contacted about websites that don't comply and they don't ever remove the HonCode), but anyway, I don't see how their principles relate to our issues? SandyGeorgia (Talk) 17:10, 28 February 2012 (UTC)

I don't know about HONcode's track record, but the principles themselves seem reasonable. I think we'll have problems with #7 ("Financial discloure"), though. Even in cases where clear financial conflicts of interest exist, we evidently have no way to address them effectively or communicate such conflicts to the reader. And worse, we're dependent on voluntary disclosure by pseudonymous editors to even learn about a financial conflict of interest in the first place. MastCell ^Talk 19:18, 28 February 2012 (UTC)

I think Sandy's right, nothing should be promoted before there's a coherent whole. But it's worth considering whether a two-tiered approach might be helpful - a shorter, to the point policy, and a longer, more detailed guideline that supports the policy and its implementation. Now, granted, knowing the way people fight over policy (i.e., by attacking it's weakest links...the supporting guidelines), the guidelines part would need just as solid consensus as the policy, prior to promotion. But it's certainly worth considering. Guettarda (talk) 18:02, 28 February 2012 (UTC)

My intention was more that we should consider the principles, not pursue their certification. Sorry I wasn't clearer. Re financial disclosure, MC's right. Because we have no way to ensure it is disclosed, we should be frank with readers that the potential of undisclosed conflict is always there.

How, precisely, would we go about "being frank with readers" on this part? A banner saying "This may have been written by drug shills?" above any pharmaceutical article? Em, I think this isn't taking us anywhere useful... Colin°^Talk 20:26, 28 February 2012 (UTC)

We already have wp:MEDICAL, it would only need a minor tweak to the wording, such as "accuracy" --> "accuracy or impartiality". LeadSongDog come howl! 20:41, 28 February 2012 (UTC)

Standards such as these require interpretation in individual cases, and interpretation will always be a matter of judgment, and in any non-trivial case, to some extent a matter of opinion. It's wrong , imo, to be exactly prescriptive in writing the standards, because there are too many individual possibilities. I do not think that this is or should be a policy; not even its parent, WP:RS is a policy, but a guideline, and this is but a specialized derivative of that. The actual policy, in the sense of something we would almost always follow, is WP:V. What happens to adopted "policies" that are not very fully supported by consensus is that they get mostly ignored,(WP:NPA is probably the prime example). the apparent agreement here is deeoptive, because the contributors here share the same general POV (one that I also share, but at least I have a skepticism about the degree to which all Wikipedians actually agree with it. ), DGG ( talk ) 18:57, 1 March 2012 (UTC)

I don't think that RS is MEDRS's "parent" page; I think that MEDRS's actual parent is WP:V, just like RS's parent page is WP:V. WhatamIdoing (talk) 23:33, 1 March 2012 (UTC)

Agree. Also I think we're losing the point here re whether page is escalated to policy. The issue was(I thought, anyway), one of enforcement, that is, can we write something that gives guidance about when text can be removed, akin to what we have in BLP. SandyGeorgia (Talk) 01:11, 2 March 2012 (UTC)

^ People are presumed to be living unless there is reason to believe otherwise. This policy does not apply to people declared dead in absentia.
^ Jimmy Wales. "WikiEN-l Zero information is preferred to misleading or false information", May 16, 2006, and May 19, 2006; Jimmy Wales. Keynote speech, Wikimania, August 2006.
^ Wikipedia:Requests for arbitration/Rachel Marsden: "WP:BLP applies to all living persons mentioned in an article"
^ Laurent, MR; Vickers, TJ (2009). "Seeking health information online: does Wikipedia matter?". J Am Med Inform Assoc. 16 (4): 471–9. doi:10.1197/jamia.M3059. PMC 2705249. PMID 19390105.

[1] People are presumed to be living unless there is reason to believe otherwise. This policy does not apply to people declared dead in absentia.

[2] Jimmy Wales. "WikiEN-l Zero information is preferred to misleading or false information", May 16, 2006, and May 19, 2006; Jimmy Wales. Keynote speech, Wikimania, August 2006.

[3] Wikipedia:Requests for arbitration/Rachel Marsden: "WP:BLP applies to all living persons mentioned in an article"

[4] Laurent, MR; Vickers, TJ (2009). "Seeking health information online: does Wikipedia matter?". J Am Med Inform Assoc. 16 (4): 471–9. doi:10.1197/jamia.M3059. PMC 2705249. PMID 19390105.

[1]

[2]

[3]

[4]