Jump to content

Wikipedia talk:Identifying reliable sources (medicine)/Archive 11

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Archive 5←Archive 9Archive 10Archive 11Archive 12Archive 13→Archive 15

Proposed modification to Scientific Consensus section of MEDRS

I would like to get the group’s feedback on a proposed addition to MEDRS. MEDRS currently covers the topics of Evidence (factual data) and Scientific Consensus (largely the interpretation of that data) in separate sections. In current practice, we commonly treat a source that is high quality for Evidence as being of equally high quality as a source for Scientific Consensus. Below I argue that this is not always true and that MEDRS should be slightly modified to recognize this.

What MEDRS currently says about Evidence

  • “Ideal sources for such content includes literature reviews or systematic reviews published in reputable medical journals, academic and professional books written by experts in the relevant field and from a respected publisher, and medical guidelines or position statements from nationally or internationally recognised expert bodies”
  • “Primary sources should generally not be used for medical content.”
  • "A primary source in medicine is one in which the authors directly participated in the research or documented their personal experiences. They examined the patients, injected the rats, filled the test tubes, or at least supervised those who did. Many, but not all, papers published in medical journals are primary sources for facts about the research and discoveries made."
  • "The best evidence comes primarily from meta-analyses of randomized controlled trials (RCTs)."

What MEDRS currently says about Scientific Consensus

  • "Wikipedia policies on the neutral point of view and not using original research demand that we present any prevailing medical or scientific consensus, which can be found in recent, authoritative review articles or in textbooks or in some forms of monographs."

Proposed modification to the section on Scientific Consensus

” Wikipedia policies on the neutral point of view and not using original research demand that we present any prevailing medical or scientific consensus. Practice guidelines issued by the National Institute of Health and Care Excellence and by major medical societies are ideal sources for information regarding current medical consensus. Other sources include textbooks, monographs, review articles and meta analyses."
” Wikipedia policies on the neutral point of view and not using original research demand that we present any prevailing medical or scientific consensus. Practice guidelines issued by the National Institute of Health and Care Excellence and by major medical societies are ideal sources for information regarding current medical consensus. Other sources include textbooks, monographs, review articles and meta analyses."

Justification

  • It is commonplace on controversial subjects (such as, until recently, statins in primary prevention) for one to be able to find dozens of meta analyses and reviews, expressing a broad range of opinions.
  • This makes it possible to cherry pick, which is one of the rationales we used for not allowing primary research citations to be used as Evidence.
  • All it takes to become a secondary source on a controversial subject is for someone to have an opinion and write a review.
  • Treatment guidelines are consensus documents written by recognized experts made up of professionals in that discipline and endorsed by societies or organizations such as AHRQ and NICE. It’s a big difference.
  • In the case of meta analyses, the authors frequently opine at great length on subjects not discussed in the literature underlying the meta analysis they have performed, such as the adequacy of the trial design, the relevance of the endpoints, and other subjects. From my POV, analyses not discussed in the literature underlying the meta analysis are primary research and should to some extent be treated as such.

Examples of how this is an issue in actual articles

  • On one controversial article that I hesitate to name lest this discussion become a referendum on the topic of that article, we have a review article written by an author with no prior publications even remotely related to the review topic. It is being used to offset the opinions of professional societies and relevant government health authorities.
  • In the article on Aripiprazole, we state in Wikipedia’s voice the that the drug “may be useful” but "it is difficult to tell" because of various issues with endpoints and trial design. These are statements of opinion by the meta analysis authors that are neither mathematically derived from the meta analysis itself nor extensively discussed in the primary literature underlying the meta analysis. Having presented the opinion of these two authors as Scientific Consensus, we then mention the positions of several professional medical organizations, one of which recommends aripiprazole as first line therapy and Level A evidence, and others of which recommend aripiprazole as one of many possible first line treatments. The current guideline recognizes the preeminent positioning of the opinion of two authors over that of several professional psychiatry groups as legitimate judgement call. The authors of the meta analysis have a total of 17 prior publications in the field between them.
  • Until recent edits by myself, the Ezetimibe article lede quoted several reviews stating that the drug should be used rarely or never, ignoring the fact that practice guidelines from most (all until the recent ACC/AHA guidelines) recommend it as second line therapy.

I’d like to emphasize that I’m not suggesting that minority opinions be edited out, but only that treatment guidelines be give some precedence in determining what mainstream opinion is. Because they are consensus documents written by large groups of acknowledged experts, the use of these guidelines as a favored source helps reduce "noise", cherry picking, and helps to assure that the opinions are those of acknowledged experts. The fact that they are issued by large specialist societies or major healthcare agencies helps assure that that they are widely endorsed and do in fact represent Scientific Consensus. Thank you in advance for your considered opinion. Formerly 98 (talk) 18:16, 18 November 2014 (UTC)

I think this makes sense. Practice guidelines and consensus statements from reputable major medical bodies are high-quality sources; ideally, we're already using them as a matter of common sense, but I don't see the harm in writing them into the guideline if there's been an issue. I agree that narrative reviews are a highly mixed bag, having written a number of them. Those found in high-quality journals and written by authors with a clear track record of expertise in the given field are good sources, but many narrative reviews don't meet those criteria. Some journals even encourage speculation or conjecture in narrative reviews, although usually under a more dignified heading such as "expert opinion". MastCell Talk 18:28, 18 November 2014 (UTC)
I also agree. I suspect most physicians rely on clinical practice guidelines and consensus statements for how they practice (rather than the random review article), and so these likely form the backbone of the actual consensus of the medical community. Yobol (talk) 18:32, 18 November 2014 (UTC)
To play devil's advocate for a moment, though, we should probably be wary of this being used as a back door to introduce publications from organizations supporting fringe specialties. (Suppose, for example, that a large organization of naturopathic physicians provided a guide to homeopathic clinical practice. We would want to be very careful about the circumstances under which that source would be used.) TenOfAllTrades(talk) 19:10, 18 November 2014 (UTC)
We can cite specific examples such as AHRQ, NICE or USPSTF as examples of the bodies that would be recognized, as well as caution against using guidelines written by advocacy groups... Yobol (talk) 19:14, 18 November 2014 (UTC)
i think this is very helpful and the definition of "secondary source" in the definitions section should also be updated to move statments by from major medical & scientific bodies to the fore. a meta-analysis might seek to overturn treatment guidelines, and what we want always is to be in the mainstream - we have no place on the cutting edge. Jytdog (talk) 23:07, 18 November 2014 (UTC)

amended proposal based on feedback above.

” Wikipedia policies on the neutral point of view and not using original research demand that we present the prevailing medical or scientific consensus. Statements and practice guidelines issued by major medical and scientific bodies national bodies (e.g NICE, AHRQ, USPTDF, ACC/AHA) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, monographs, review articles and meta analyses. Statements by advocacy groups are generally not useful sources." Jytdog (talk) 23:18, 18 November 2014 (UTC
some typos in that. How about
Wikipedia policies on the neutral point of view and not using original research demand that we present the prevailing medical or scientific consensus. Statements and practice guidelines issued by major professional medical societies (For example, the American Diabetes Association, the European Society of Cardiology or the Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example,AHRQ, NICE, USPTDF, and WHO) (scientific bodies national bodies (e.g NICE, AHRQ, USPTDF, ACC/AHA) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, monographs, review articles and meta analyses. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting. Statements by advocacy groups are generally not useful sources." (my additions underlined) Formerly 98 (talk) 00:01, 19 November 2014 (UTC)
OK by me! Jytdog (talk) 00:34, 19 November 2014 (UTC)

I would also add the Cochrane collaboration to that list of scientific bodies and than it has my support. Is this replacing some text or simply being added? Doc James (talk · contribs · email) 01:09, 19 November 2014 (UTC)

i struggle with adding the Cochrane Collaboration as a major medical or scientific body. Would you please explain why you put that volunteer association (somewhat akin to WP) on par with NICE? The meta-analyses they produce are of course great secondary sources. Thanks. Jytdog (talk) 01:31, 19 November 2014 (UTC)
Aha, this is the stickiing point. I think the Cochrane group is the first tier for Evidence. But they are often far, far from Scientific Consensus, and are often seeking to influence it rather than representing it. Some examples:
  • The Cochrane group's position of the influenza vaccine is quite the antipode of what is recommended by the IDSA, other infectious disease groups, and virtually every ID doc I interact with. Most hospitals seem more concerned with enforcing mandatory vaccination of healthcare workers than with Thomas Jefferson's opinion on the flu vaccine. They may be right, but they are not Scientific Consensus here.
  • A cited above, the position of the Cochrane group on antipsychotics is often well outside the position not only of professional psychiatric organizations but also NICE. When they calculate an effect size from the results of 15 historical clinical trials of aripiprazole, they are a secondary source and the very best secondary source for Evidence. When they opine about whether the endpoints or trial designs were adequate, I see that as primary research. They may be right, but there is no evidence that their conclusions are Scientific Consensus.
  • I can provide more examples, but my wife wants me to run out and pick up some Indian Take out.

It may be that I am slicing things a bit too thinly here by separating out Evidence (data) and opinion on what that data means and how it should be used (my understanding of what Scientific Consensus is). But the problem I keep coming back to is two authors who are fairly junior researchers expressing opinions that did not derive directly from mathematical analysis of the historical trial record, and these opinions being used to rebut or offset consensus treatment guidelines endorsed by professional medical specialty societies. They have a right to their opinions, they may be right. But while they seek to influence Scientific Consensus, they don't necessarily represent it. Formerly 98 (talk) 01:38, 19 November 2014 (UTC)

I am of the position that both the best available evidence and the positions of major medical organizations should be emphasized under medical uses / treatments / prevention. Most of the time both are similar and in the few situations when they are not these different positions should simply be listed. The Cochrane collaboration's reviews do have a significant impact on many peoples clinical practice similar to that of position statements of nationally and internationally recognized bodies thus my position for inclusion. They are performed using a specific methodology and are independent from what is being studied. Both of these are important. Doc James (talk · contribs · email) 03:44, 19 November 2014 (UTC)
Cochrane is always an important voice. I would support always mentioning their position, whether it is aligned with the viewpoints of the aforementioned groups or not. But NPOV requires that we present different positions according to their prevalence among experts in the field. In the case of treatment guidelines from major medical societies, we have evidence that they are widely supported by experts because they are endorsed by the medical societies that formed the committees that drafted them. In the case of Cochrane, we have no direct evidence as to how widely accepted their views are on any subject except by reference to other reviews.
I'm not intending to trivialize or minimize Cochrane in any way. But their role is to investigate and present positions, not to summarize consensus. They are in arguable playing their most important role when they challenge consensus. Its just a different role. I just don't think NPOV allows us to give a preeminence to Cochrane's position when it conflicts with the consensus treatment guidelines of multiple medical groups. In that case they become an important minority voice and get the weight that NPOV assigns to important minority voices. Formerly 98 (talk) 04:06, 19 November 2014 (UTC)

They are an important position in medicine. And I believe that they along with other major medical groups should be presented prominently. That if say the USPSTF or the American Radiology Association disagree with Cochrane's position on breast cancer screening does not mean that we should remove Cochrane's position. I also defend the presense of both the position of the USPSTF and the ARA by the way.
If Cochrane should only be used when other major organizations agree with it is more or less saying we should not use the Cochrane reviews. So I guess I oppose your suggested change. Doc James (talk · contribs · email) 04:24, 19 November 2014 (UTC)

funny as I wrote User:Casliber/Fivecrat in November 2008.

You have lost me... Doc James (talk · contribs · email) 06:00, 19 November 2014 (UTC)
I agree with the change for the reasons outlined by F98, but I agree with Doc James - Cochrane are known for their rigour and neutrality and held in high esteem by doctors worldwide (neutrality being important when one reads how many trials have issues and then get referenced in review papers, creating a woozle effect as the original errors get buried in numerous on-citing and "validity". It is only when one appreciates how widespread this is that one appreciates how important cochrane is. Cas Liber (talk · contribs) 05:35, 19 November 2014 (UTC)
  • A minor quibble with the wording: "Wikipedia policies on the neutral point of view and not using original research..." implies incorrectly that using original research is precluded by policy, but that is not at all the case. WP:NOR precludes wp from publishing OR, not from citing it. Of course MEDRS should continue to exclude or restrict the citation of primary sources, but policy does not do that. LeadSongDog come howl! 04:52, 19 November 2014 (UTC)

@Doc James: I personally think Cochrane is always an important voice, and would always mention their position on any subject they have covered. But when they disagree with the consensus treatment guidelines that have been endorsed by the 84,000 member APA, they become an important minority viewpoint, and consequently get less weight than the majority opinion. The APA makes a point of generating consensus guidelines. The Cochrane groups's job is not to generate consensus documents, but to investigate and make recommendations. They are in fact at their most valuable when they challenge the current consensus/status quo.

Thats what I think. If you disagree, your thoughts on how your position fits with WP:NPOV when the Cochrane position is in the minority would be helpful here I think. Formerly 98 (talk) 05:36, 19 November 2014 (UTC)

Going back to the issue with breast cancer screening, I do not agree to excluding Cochrane just because the ARA says breast cancer screening is good. Same thing with the APA, we have position statements that have varying degrees of evidence based-ness. In this situation we attribute the position of the APA and that of the Cochrane review when they disagree. I believe this is WP:NPOV by providing the positions of all high quality sources. Doc James (talk · contribs · email) 05:49, 19 November 2014 (UTC)

I absolutely agree on presenting all. I'm just trying to get to weight. Formerly 98 (talk) 05:55, 19 November 2014 (UTC) My understanding is that NPOV requires weighting according to prevealence of the viewpoint among experts. Is this not correct? — Preceding unsigned comment added by Formerly 98 (talk • contribs) 05:56, 19 November 2014

American Radiology Association, per above. --Anthonyhcole (talk · contribs · email) 09:47, 19 November 2014 (UTC)

I think the second draft above is a real improvement on the current wording. "Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting" ensures that contradictory Cochrane conclusions won't be muscled out. Regarding weight, if one of their conclusions challenges the existing consensus, I hope we'd give it enough prominence to be sure the reader won't miss it. --Anthonyhcole (talk · contribs · email) 10:25, 19 November 2014 (UTC)

I have lost faith in Cochrane after seeing what a disaster they made of their review of the ketogenic diet (see Talk:Ketogenic diet#Incorporating 2012 systematic review). The introduction to that review, covering history, makes a number of factual errors. Their analysis and summary of the studies is flawed and very badly presented. The authors, from what I can see, are not experts in the subject and work in local district hospitals (one wonders if they actually have any personal experience of that treatment since their hospitals would be unlikely to offer it). Cochrane's formulaic approach may work when there is sufficient data or when the authors are wise enough to admit they have nothing useful to say. Here is an example of authors over extending themselves and producing a paper that is only fit to line a hamster's cage. The difference with a body of experts (or a combination of experts + other parties such as patient groups such as we see with SIGN) producing evidence-based treatment guidelines is that they are required to produce a document that has some utility. They aren't engaging in a mechanical process, ticking the systematic review check boxes, and churning out something of no practical use. And not all aspects of treatment (drugs, protocol, review) are studied by RCTs so there is still a place for expert consensus opinion -- something Cochrane's work cannot supply.

I think there is a danger that our "treatment" sections become OR when we rely on academic papers that do not actually describe current treatment nor have any authority behind their opinions to guide ideal treatment. We are all fans here of evidence based medicine, but there is a danger our enthusiasm for this leads to a gap with reality. Our readers need to learn how diseases are actually or at least expected to be treated alongside learning about the strength of the evidence behind these treatment options. Therefore I do not support Cochrane being given equal status to official or consensus treatment guidelines for sourcing our "treatment" sections. Btw SIGN is the Scottish equivalent of NICE (England + Wales) and their work is a useful online source. -- Colin°Talk 10:39, 19 November 2014 (UTC)

Some feel we should emphasis meta analysis others feel we should emphasis position statements. Both can have issues and be of low quality or out of date. At this level one is often splitting hairs. The bigger issue is the loading of articles with primary sources. If this was addressed and all our medical content was based on high quality meta analysis or position statements of nationally or internationally recognized organizations than I think it would be a useful conversation. Doc James (talk · contribs · email) 15:51, 19 November 2014 (UTC)

Cochrane is widely considered one of the best available sources for evidence. This is not to say they're perfect, because "best" is a relative and not absolute term, but they rank at or near the top for evaluation of the current state of evidence and what conclusions can be drawn. Medical associations are as well imperfect, and their positions can be subject to influence from factors other than objective evaluation of the evidence, such as politics. This is a strong reason why Cochrane's position as an independent organization must not be devalued. Zad68 16:14, 19 November 2014 (UTC)

I concur with Zad directly above and with Doc James both directly above and in general. While any source may have limitations or failures Cochrane is one of the best. Various organizations are subject to influences and as an encyclopedia while the positions of these organizations carry substantial weight, evidence and neutrality are most important. - - MrBill3 (talk) 16:41, 19 November 2014 (UTC)
I'm late to this discussion and the thread is a jumble, but I support the general gist of the proposal with regards to giving primacy to existing professional guidance from authoritative bodies.
However, I too support Cochrane systematic reviews, even when they contradict existing guidance. They Cochrane Collaboration have rigorous standards, it is about as impartial as you can get, and steers clear of expert opinion for the sake of it.
We should also allow inclusion of dissent with guidelines. The NICE guidelines for osteoporosis, for instance, have been vigorously criticised by other bodies.
I don't think I agree completely with the statement that we should never quote the conclusions of meta-analysts. If a certain widely used treatment has been widely studied but studies are conflicting and methodologically divergent (for which there are objective standards such as CONSORT), it might be appropriate to explain this even to the non-initiated reader. JFW | T@lk 17:00, 19 November 2014 (UTC)
I think the important thing to recognize about the Cochrane Library is that they represent an extreme when it comes to evidence-based medicine (an extreme satirized in articles such as Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials). Cochrane reviews tend to address questions from an abstract, purist perspective and to discount practical considerations where concrete evidence is lacking. On the other hand, clinical medicine is usually messy and most situations don't lend themselves well to this approach. Clinical-practice guidelines are useful because they are practical, and they fill in the gaps where evidence is lacking, usually with a consensus of expert opinion. I think that ideally we'd use the Cochrane/purist approach together with the practical guideline approach, where both exist. MastCell Talk 17:28, 19 November 2014 (UTC)

Thank you everyone for the input. I hear the concerns about the value of Cochrane and would like to repeat that I never suggested that we include only a statement from a single group in the articles or that we exclude Cochrane, and the proposed language does not suggest this in any way. I do believe that MEDRS should reflect WP:NPOV which is a pillar that cannot be modified or abrogated by the Projects. WP:NPOV requires that we weight viewpoint based on their prominence, and not on our personal opinions as to which source is "best".

To address these concerns, I suggest modifying the language as follows. "Wikipedia policies on the neutral point of view and not using primary research demand that we present the prevailing medical or scientific consensus. Statements and practice guidelines issued by major professional medical societies (For example, the American Diabetes Association, the European Society of Cardiology or the Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example,AHRQ, NICE, USPTDF, and WHO) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, monographs, review articles and meta analyses. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting. Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the Cochrane Collaboration).

Thanks all again. I hope this will reduce some concerns and look forward to hearing your thoughts. Formerly 98 (talk) 17:18, 19 November 2014 (UTC)

I think this would be reasonable. MastCell Talk 17:28, 19 November 2014 (UTC)
It's progress, but not there yet. The "internationally recognized and influential groups" could be taken to include OPEC, the Vatican, or the Hell's Angels. Their relevance or impartiality in relation to biomedical topics may not exactly make them the most reliable of sources. Also, as I said above, "not using primary research" should be "not publishing primary research".LeadSongDog come howl! 17:44, 19 November 2014 (UTC)
I don't think we need to worry too much about deliberate misinterpretation of the language that would be shot down in a heartbeat in an RFC, but we can certainly add the word "medical". MEDRS as currently constituted all but completely prohibits using primary research and WP:RS discourages it, so I think that part is OK. Formerly 98 (talk) 18:14, 19 November 2014 (UTC)
Is that a typo in your lead sentence: should say "original research". Freudian slip? There are no policies on "not using primary research". The "other sources" sentence already includes the Cochrane library ("meta analyses") therefore I see no benefit to the bold text addition and an awful lot of trouble. As noted above "internationally recognized and influential groups" is deeply problematic. "Care should be taken to highlight" seems to be asking editors to find an argument and emphasise it. The phrase "evidence-based" is not Wikipedia policy. I do wonder sometimes if "efficacy" is all that some editors care about. Look in any of our major diseases articles and often the first thing you will find in the treatment or management section is a statement about which drugs are effective backed by a Cochrane review. Yet Cochrane reviews purpose is not to describe how diseases are treated nor are the opinions of their authors in any shape or form a source for the "prevailing medical or scientific consensus" on how a disease should be treated. There is remarkably little in healthcare that can be learned from randomised controlled trials. So I oppose the addition of the bold text. -- Colin°Talk 18:57, 19 November 2014 (UTC)
Well, there is some conflation going on. In order to construe a primary source as useful, an editor implicitly engages in wp:SYN, which is a form of original research to create a secondary publication. The prohibition of SYN is policy driven, but MEDRS has only risen to the status of guidance in the interpretation of other policies. It might almost be cleaner to wikilink the policies and avoid the assertions entirely. LeadSongDog come howl! 20:40, 19 November 2014 (UTC)

On Cochrane

MastCell's parachute article is funny but there is a danger that parody is not taken seriously. Let's pick a real example, and I'm afraid it has to remain on the topic of epilepsy since that's something I can actually talk about, albeit with my small lay brain. The first thing our epilepsy article does when mentioning specific drugs is to comment on their efficacy. It cites three Cochrane reviews. The article does go on to list practice guidelines, citing NICE, but the important thing is that three Cochrane reviews are cited and used before anything else. So the first thing a reader wanting to know about the drug treatment for epilepsy learns is that "Phenytoin, carbamazepine and valproate appear to be equally effective in both focal and generalized seizures. Controlled release carbamazepine appears to work as well as immediate release carbamazepine, and may have fewer side effects." This obsession with discussing efficacy trial results permeates our medical articles, and comes from the fact that we over-laud systematic reviews (and others over-use primary research papers) as though our articles should be founded upon them.

So let's consider if one cited Cochrane review actually supports the statement "appear to be equally effective in both focal and generalized seizures" for phenytoin and carbamazepine. The paper is PMID 12076427 and the full text "Carbamazepine versus phenytoin monotherapy for epilepsy". Firstly, the review only looked at the drugs "when used as monotherapy in subjects with partial onset seizures, or generalized onset tonic-clonic seizures with or without other generalized seizure types." -- which is not all "generalized seizures" (e.g. absence seizures and atonic seizures are different) and not all patients are treated with monotherapy. Secondly, consider the primary outcome is "Time to withdrawal of allocated treatment". This is one recommended measure of epilepsy treatment that attempts to balance whether the drug is effective at reducing seizures and whether the side effects are tolerable. However it is a proxy for effectiveness and side-effects and not without its problems. Still, have a good read of the article. As you read, your heart should sink as study after study is thrown away, data is found to be missing, studies looked at different endpoints from this one's, or studies had methods that aren't what we'd wish for, or only examined seizure types that aren't as general as the author's wanted. In the end, the authors decided not to use the studies' findings but to look at the individual patient data where they could obtain it. They got this for just three trials. One trial looked at children and two looked at adults. One trial looked at partial onset seizures only and the other two at patients with either partial onset seizures or generalized tonic-clonic seizures. These are not patient groups that are typically combined for analysis (heterogeneous). Only one was double-blinded. Is "buggered if I know" a fair summary of the review conclusion?

The limitations of their proxy for efficacy+tolerance become apparent when you read the background section of the paper. The experts have already concluded that these drugs are much of a muchness wrt reducing seizures and are more concerned with their side effect profiles. Are these side effects the kind that influence patients asking to stop a medication during a trial? No. Phenytoin has long term problems with facial disfigurement. Both drugs may very rarely kill their patients with a nasty alergic rash. Carbamazepine may cause neural tube defects in babies, though neither drug is without its pregnancy issues. None of these concerns that the author say influence drug choice in clinical practice are ones that a short-term efficacy study of 50 patients is likely to discover.

We have a Cochrane review that has discovered aspects of our ignorance but otherwise does not support the statement that these drugs are equally effective, nor does it even attempt to answer the implied statement that both these drugs are effective.

Before anyone trumpets or recommends Cochrane or indeed any form of literature or sourcing, one must first consider what statement one is trying to say in article text. I recommend that for disease treatment/management sections, the primary focus should be to source information that describes what the current treatment is and what our best expert minds consider is the best clinical practice. For drugs, the primary consideration is for what the drug is licensed; for what it is a recommended treatment by authorities; and for what it might be widely used even if lacking evidence or in the face of disapproval (e.g. cough medicine). An academic paper that effectively says "we cannot answer the question" is not a source for very much at all. -- Colin°Talk 23:11, 19 November 2014 (UTC)

Colin, is this really a fault with Cochrane in particular? Or just that a Wikipedia editor didn't do great job in summarizing the source accurately, which can happen with any source? I think it's the latter, and should be corrected in the article. Zad68 16:23, 20 November 2014 (UTC)
It is both. But I think this sort of misuse of Cochrane is common. Reviewers look at the article text, see it is cited to Cochrane, apply their prejudice that "Cochrane is top tier for evidence" and assume everything is fine. But also, this isn't about the article misrepresenting a paper (though that is bad enough) but that the paper should never have been used on WP at all -- if you think the article should be rewritten to reflect what that paper concluded, then you really haven't understood my point at all. None of our readers care that some non-expert volunteers conducted a dry academic exercise, and were so frustrated at the lack of primary published material upon which to build that they reached into the studies' source data to conduct primary research themselves... and still they had to admit defeat -- that they had learned nothing. I do believe some editors are seeking out Cochrane and other systematic reviews as a foundation upon which to build an article. You write articles by reading widely on the subject to become familiar with the topic and the literature, and then deciding what each section in your article needs to say and what reader questions you need to answer. Only then do you select the sources with the best power to provide that text. -- Colin°Talk 18:44, 20 November 2014 (UTC)

I've been stewing on this Cochrane issue for a while now. I know we have a formal collaboration with them and that is a good thing, for sure! But i have concerns similar to those of Formerly and Colin. The specific proposal by Doc James above, was to add "Cochrane Collaboration" to the list of sample major scientific/medical organizations. I for one struggle with listing it as an organization for several reasons:

  1. as an organization I don't believe it issues statements. It publishes meta-analyses. We already mention those. We could add something like "meta-analyses, such as those generated by the Cochrane Collaboration"
  2. as far as I can see, the Cochrane Collaboration is a very different animal from something like NICE or the American College of Cardiology. The latter two have specific remits - NICE for evaluating treatments on behalf of NHS, ACC for certifying cardiologists and overseeing that field, including issuing treatment guidelines. Both of those take stances that directly affect clinical care. CC has pretty much one reason to exist - facilitate production of meta-analyses.
  3. and here is the kicker. The conclusions of a given CC meta-analysis may or may not be adopted by the mainstream. WP is not out to WP:RIGHTGREATWRONGS or work to change clinical care. We are not on the vanguard of anything, including the science-based medicine movement. We present the mainstream consensus. here is a somwhat perhaps controversial proposal: If a new meta-analysis article authored under the CC guidelines and included in their database, produces conclusions that recommend changing what clinicians do or that contradicts mainstream clinical practice, that meta-analysis publication is a WP:PRIMARY source for that conclusion, and we wait to see if the conclusions are actually adopted by the mainstream and are, for example, included in treatment guidelines or we get other evidence that they have actually been adopted. (!) This would fit with Colin's objections as well.

So, no we should not list CC as an example organization with NICE, AHRQ, etc. yes it would be great to name them as a producer of meta-analyses, as they are the premier generator of them. that's my view on the question at hand. Thoughts?? Jytdog (talk) 16:31, 20 November 2014 (UTC)

I think Jytdog presents a valid consideration. I think attribution resolves much of this, particularly in regards to recommendations/implications for clinical practice. I think Cochrane is the premier source for evidence. An encyclopedia should reflect the evidence itself as well as the interpretation and implementation of the evidence. I think due to the concerns of bias/influences raised earlier the positions, recommendations and standards of care from various organizations should be clearly attributed. - - MrBill3 (talk) 17:42, 20 November 2014 (UTC)
(Side note: An encyclopedia should reflect reality, to the extent that this is possible. Unfortunately, the closest we get many times is reflecting the currently published evidence, which often differs from reality.) WhatamIdoing (talk) 04:21, 26 November 2014 (UTC)
Evidence-based methods are effective, and effective methods should be evidence-based. If a method appears to be effective, then it should be possible to prove it. If the research has not been done yet, it should be. We must remember that "Absence of proof is not the same as the absence of fact; it simply demonstrates the lack of adequate research." - Robert Sydenham. "Lack of evidence in the literature is not evidence of lack of effectiveness." Until that research is done, claims of effectiveness are uncertain, unusual claims of effectiveness doubly so, and consequently the marketing of products and practicing of methods based on such uncertain, unusual, and undocumented claims may be unethical, possibly dangerous, and often illegal.
Wikipedia is not a crystal ball. We are supposed to always be "behind the curve". We document what has been proven in science and medicine. Other claims of "reality" are presented as beliefs. Until the proof is in, we can't know that it really is "reality". Unfalsifiable claims aren't necessarily "untrue", but one cannot know whether or not they are true until evidence arrives. We wait for the publication of that evidence in RS before we write it here. We are always waiting around the bend, and behind the curve. We must never get ahead of the curve. That's the land of original research. BullRangifer 04:44, 26 November 2014 (UTC)
BullRangifer, this is a rather extreme view fit only for the land of theory. It does not reflect ethics or practicality, nor our policy. If a consensus of experts are convinced of the utility of a treatment then that is enough for us: so if our best sources agree "x is an useful treatment for y" then we report that with the same level of confidence as they do. If they have doubts then we document them. If we have doubts, nobody cares. As with the parachute example, there are just too many times where complete scientific certainty would prevent a pragmatic approach this is necessary in the real world. Sure sometimes the real world makes a mistake, but then the scientific approach is also limited. One cannot do an RCT that will discover your drug may kill 1:2000 patients with liver failure. To demand that research to prove all treatments be done to the highest gold-quality trial standards, is unhelpful and unrealistic. Once the evidence is good enough to convince those making guidelines, then anything more is simply a bonus. It may, at that point, be unethical to conduct placebo controlled trials, or simply that in our world of limited budgets, there are better things to spend our money on. For example, the Neal RCT on the Ketogenic diet (PMID 18456557) could not have a typical control group as it would be unethical to withhold treatment (already) believed to be effective. The fact that a control group could be created was an artifact of the waiting list caused by under-funding of the NHS. The Cochrane paper states that Neal should be repeated. That is ivory tower thinking. It won't be. -- Colin°Talk 12:45, 27 November 2014 (UTC)
There is a real danger that because Cochrane is all about evidence based medicine, and many assume that is a good thing, that it necessarily produces something valuable to our encyclopaedia. I really want us to question this belief. Firstly on the ketogenic diet paper, we have a deeply flawed paper written by two local hospital doctors who have (I strongly believe) no experience in treating refractory epilepsy in children let alone any experience with the diet. How can this be considered a reliable source used to make treatment recommendations in our epilepsy article? Without the Cochrane name, this paper would not have been published I am sure. With the phenytoin vs carbamazepine paper, we see a methodology that is so rigid that is incapable of producing a paper worth reading let alone citing. Without the Cochrane name, would this non-result paper be published? Other organisations and writers are capable of reviewing those (and other) drugs and coming to a conclusion that has some merit and practical significance in the real world. And I fully agree with the concerns that a Cochrane paper may end up being a primary research paper -- particularly when as with the phenytoin vs carbamazepine paper, it performs novel analysis of raw unpublished data. -- Colin°Talk 18:44, 20 November 2014 (UTC)

How much does NICE or the AHRQ or the APA dictate practice? These are just guidelines that physicians may follow to various degrees. They also may or may not be adopted by the mainstream. Look at antibiotics for the acute bronchitis. Little evidence supports their use. No guidelines support their use. Yet they are commonly used. We basically have three things 1) what the evidence shows 2) what the guidelines say 3) what physicians do. Cochrane reviews are not primary research papers. Doc James (talk · contribs · email) 20:35, 20 November 2014 (UTC)

James, that epilepsy paper took raw unpublished patient details and conducted novel research on it. That's not a secondary source, nor is it a review (systematic or otherwise) of the literature. The crossed the line, in an effort to get some data they could use to answer the question. And still failed. And the ketogenic diet paper is written by non-experts who made mistakes, so I really do have to question what their criteria is for selecting authors and what review their work goes through. But I agree that if antibiotics are commonly used for the acute bronchitis then a good source for the article subject should mention this -- this is why I think professional books are perhaps more useful than research papers and reviews of such -- because such a book looks at the whole topic rather than trying to write something novel. -- Colin°Talk 18:51, 21 November 2014 (UTC)
I suppose we can get into some pretty serious wikilawyering here, but I think there is a major problem with that example James. ID docs, who are arguably the "experts" whose opinions we are seeking in determining Scientific Consensus on treatment of infectious diseases, take the antibiotic guidelines very, very seriously. The abuse you are referring to is mainly by ER docs and primary care docs, whose practice is gradually being dragged into guidance compliance by ID doc led antibiotic stewardship programs. Formerly 98 (talk) 22:31, 20 November 2014 (UTC)
Colin, I'm not sure I see a problem with that epilepsy paper (other than perhaps MEDDATE). The authors are quite direct about their methods. Yes, they did their own analysis, but not their own observations. The analysis was systematic and at least in principle it should be reproducible by anyone who cares to do so. They drew the rather commonplace conclusion that the original studies, even pooled, were too small to find a significant difference between the two interventions (the 95% confidence intervals overlapped, and they spanned from <1.0 to >1.0 in each respect considered). I don't see that as a failure of the review, just a finding that neither of the two interventions examined is so much better than the other as to be measurable with the trial designs and populations used. Am I missing something? Are there reliable contradictory sources? LeadSongDog come howl! 19:45, 21 November 2014 (UTC)
LeadSongDog, we like reviews because they are secondary sources -- they are another look at already-published work. We like systematic reviews because they hopefully take some biases out of the process, and meta analysis can be a powerful way of increasing the power of several trials. But that isn't what happened in that paper. Firstly, they admitted that the differences in efficacy were likely to be small but differences in side-effect profile were not -- but implied that the clinical decisions upon these were being made without good evidence. So they then went to produce good evidence. For the side effects, they used a proxy that was never going to discover the long-term or rare-but-serious side effects that currently influence drug choice. And then they went and did their own original research on the patient records. By "not their own observations" I'm not clear what you mean. Epilepsy trials do not involve doctors counting seizures. Instead, they rely on patients or their carers doing the observations and submitting a seizure diary. So really they went back to the same raw data that the original trails collated. And pouring over already-collected raw data, looking for patterns, is a well known dangerous path. Their analysis of that data was no more "systematic" than the analysis one would do in any original research. [It is interesting to note that the reasons many of the studies they first selected are then rejected is because they had the same patient-set as another study -- which is all this paper ended up doing]. One problem with the Cochrane approach is they throw away so much that they/we then over-magnify the significance of what they found (or failed to find). You can't say the drugs are equally effective (which is what our article does and is the most serious problem). All one could state is a fact concerning our ignorance when someone performs a rather academic method on existing trial data. This on its own is of little importance to an encyclopedia -- there are lots of things we are ignorant about, so why pick the things that happen to have a Cochrane paper behind them? My concern is that our RS guideline might lead (and clearly does lead some) editors into starting with a Cochrane paper and then trying to find a way of including the results into our articles. Or nonsense about "Cochrane reviews are not primary research papers." The Cochrane methodology can be a way of finding first class evidence. But sometimes, it produces something that isn't of itself of any importance. And sometimes, as with the KD paper, it is written by authors who do not deserve the authority and reverence we seem to be giving to all Cochrane reviews. -- Colin°Talk 15:45, 24 November 2014 (UTC)
Without wishing to enter into any fray here, I think it may be worth bearing in mind that Cochrane reviews are generally produced only when there's a real clinical need to clarify the evidence regarding a highly specific clinical question. So it might be argued that every Cochrane review is likely to be relevant to us somewhere or other. At the same time, yes, surely we should maintain editorial judgement always... (as in "not transcription monkeys"!). And we shouldn't be expecting meta-analyses and systematic reviews to provide a broad view of a topic that one might look for in an ordinary review; that's because those sorts of highly-structured papers are obliged to keep a narrow focus around the particular research question (see, for example, how tight the PRISMA checklist is [1]). In other words, from our editorial perspective systematic reviews/meta-analyses and more general reviews serve different purposes, and we need to use both, imo, as appropriate: ordinary reviews are especially valuable for descriptive content, whereas the best quality systematic reviews and meta-analyses are the ones to use, when available, to source information regarding specific clinical questions regarding best evidence. And, personally, I feel that evidence-based clinical guides such as NCI's PDQ and NICE's CKS (not strictly speaking "practice guidelines", I think) also deserve mentioning here with their somewhat broader focus. Certainly, we're unlikely to incorporate highly relevant considerations about patient choice etc if we only turn to systematic reviews and meta-analyses. 109.157.83.50 (talk) 19:36, 24 November 2014 (UTC)109.157.83.50 (talk) 18:59, 24 November 2014 (UTC)

implementing

ok, i took a shot at implementing the proposal. Here is the change to the original language:

Wikipedia policies on the neutral point of view and not using original research demand that we present any prevailing medical or scientific consensus, which can be found in recent, authoritative review articles or in textbooks or in some forms of monographs. Statements and practice guidelines issued by major professional medical or scientific societies (For example, the American Diabetes Association, the European Society of Cardiology, or the Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example, AHRQ, USPSTF, NICE, and WHO) are ideal sources for information regarding current medical and scientific consensus. Other sources include textbooks, some forms of monographs, review articles, and systematic reviews that include meta-analysis. Significant minority opinions taken from reliable secondary sources should also be presented with appropriate weighting. Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the Cochrane Collaboration). Although significant-minority views are welcome in Wikipedia, such views must be presented in the context of their acceptance by experts in the field. Additionally, the views of tiny minorities need not be reported.

Hopefully this will fly... Jytdog (talk) 20:37, 23 November 2014 (UTC)

This strikes me as fairly close to the middle of the range of opinions expressed above, and addresses my concerns that led me to raise this issue, so I support. Formerly 98 (talk) 21:16, 23 November 2014 (UTC)
  • Hello, I'm speaking in my role as the Wikipedian in Residence at Cochrane Collaboration. As the liaison between the Wikimedia Foundation projects and Cochrane Collaboration I've been following this discussion, and I find it fascinating as an example of bottom up way that policy and guidelines are made on Wikipedia English. :-) Cochrane is keenly interested in understanding the various ways that Wikipedians want to use the Cochrane content. While this discussion doesn't change much in the approach that we plan to use to help improve the quality of content and references, it does give some extra clarity and guidance to the health topic experts and researcher who are affiliated with Cochrane and want to assist with disseminating the finding of Cochrane Reviews into medical articles. Thanks to everyone for participating in the discussion. Sydney Poore/FloNight♥♥♥♥ 20:13, 24 November 2014 (UTC)
Hmm, I'm the bottom??? :/) Formerly 98 (talk) 21:44, 24 November 2014 (UTC)
I tweaked the wording a bit; as noted above, I found the text suggesting we "highlight" specific content and the too vague (IMO) discussion of "influential" groups to be a step backward. I hope this meets with approval. Yobol (talk) 00:57, 25 November 2014 (UTC)
No offense, but I reverted this. My sense of the middle of the road view from the discussion above was that we had agreement that treatment guidelines from recognized medical groups were a preferred source for determining Scientific Consensus, but that a slight plurality felt that Cochrane's views always deserved mention even when they were out of synch with consensus documents of this type. The language that was added, "care should be taken to appropriately analyze the entire body of reliable sources on the topic, as well as the quality and quantity of sources that support each view, to determine if a consensus is present and how much emphasis to give each viewpoint", completely negates a central point that 80 or 90 percent of the commentators agreed on. It takes us back to every chemist who decides to write a review article on thoracic surgery is a reliable source whose opinion can be used to offset that of the National Institute of Health and Care Excellence. Formerly 98 (talk) 01:50, 25 November 2014 (UTC)
I have significant reservations that we are encouraging people to "highlight" specific wording from "internationally recognized and influential groups" which is, frankly, ridiculously vague. Note that wording as stated not only encourages us to highlight Cochrane views, but views from this vague group that includes Cochrane. This will lead to disputes in the future as to what is an "influential group"; novice editors here will no doubt have a very different view of what fits in this group than veteran medical editors. From my standpoint, this sentence alone makes the implemented edit undesirable, and if we cannot find a way to improve it soon, I will restore the previous version until we can fix this problem. Yobol (talk) 02:07, 25 November 2014 (UTC)
Strongly oppose this wording and agree with Yobol. I think you are trying to achieve too much. We can indicate which sources are good for which things (something I'm trying to emphasise in the above discussion -- that merely saying "there sources are great" leads to people misusing them). But I don't think it is our business as an RS guideline how to settle NPOV disagreements. The text starting "Significant minority opinions" sentence is mostly standard policy of which there is nothing medical to add -- so let's not repeat it. But the sentence "Care should be taken to highlight evidence-based challenges to consensus views originating from internationally recognized and influential groups (for example, the Cochrane Collaboration)." has no policy foundation whatsoever and is an appeal to authority -- just because they can publish influential papers doesn't mean all their work is good or influential. It also has the language of the battleground ("challenges to consensus") whereas some of the text in a Cochrane paper may be true but to repeat them would mislead. Further up, someone asked me if there were reliable sources that disagree with the epilepsy paper. Again, that's battleground talk. One shouldn't have to find an opposing view in print in order to make the generally editorial judgement we all face when building an article. And it should be a building process, not a random collation process.
Let me give an example, returning again from to the KD. The Cochrane review highlights how patients don't often stick with the diet for many years. There are several reasons for this but to repeat this fact alone can give the impression that this is not really an effective treatment (i.e., one that works only briefly or one that is so intolerable that nobody can put up with it). But in the context of a child with refractory epilepsy, who is unsuitable for surgery, the alternative is to try yet another drug after already trying several. If I tell you the chance of that other drug making a difference is about 10%, you'll have an idea of the drop out rate for that option too. So a neutral article would look at treatment options for children with refractory epilepsy and find the best sources to discuss that area. It would not start with a systematic review of just one treatment and randomly copy over whatever factoids are felt to be interesting. That is why I think the lauding of Cochrane papers is so dangerous from an article-building point of view. The facts (and ignorance) may be reliable but are often so specific to the rigorous method used or so narrow focused, that repeating them gives a very unbalanced article. It isn't about balancing "significant minority viewpoints" or "challenging" opinions all the time -- usually it is a mundane but vital business of working what to say. Something that will come naturally if one fully studies the literature, including those hard bound things containing hundreds of pages that don't require an internet connection. -- Colin°Talk 12:55, 25 November 2014 (UTC)

Yup likely trying to achieve too much. Reviews by Cochrane are of the same caliber as those from AHRQ and USPSTF. They often represent a more international perspective while AHRQ and the USPSTF represent a more American POV. Have edited to remove this device. Doc James (talk · contribs · email) 04:18, 28 November 2014 (UTC)

I am not seeing consensus for these changes thus restored it to the previous version. Doc James (talk · contribs · email) 04:35, 28 November 2014 (UTC)
My impression was that there was a large majority that agreed that the Cochrane analyses were less representative of scientific consensus than the aforementioned treatment guidelines, with mainly yourself and one or two others in dissent. I saw the disagreement as mainly between those who thought that the Cochrane group's analysis deserved special mention (as an important minority view) when it differed from this consensus view, and those who did not think it deserved special mention.
How to resolve? Formerly 98 (talk) 05:38, 28 November 2014 (UTC)
There appears to be a couple of people who are not impress with Cochrane reviews. They however appear to be the minority. The proposed wording seems to separate out Cochrane reviews as being less "good" than those by AHRQ for which I do not see majority support or consensus. Doc James (talk · contribs · email) 05:43, 28 November 2014 (UTC)
doc james just added content that placed cochrane on par with guidelines etc in this diff. I reverted, as I do not see consensus in the discussion above for putting findings of meta-analyses on par with treatment guidelines. more thoughts on this are appreciated. Jytdog (talk) 07:16, 28 November 2014 (UTC)
Agency for Healthcare Research and Quality contracts and produces systematic reviews and meta analysis such as this one [2]. It does not produce guidelines or policies. USPSTF also comes out with recommendations based on meta analysis not policies or position statements. The USPSTF is underneath AHRQ.
Cochrane of course is a major scientific society and is a quasi-governmental health authority with a seat at the World Health Assembly.[3] To further expand upon this the World Health Organization is an International Governemental Organization at which Cochrane has input. Doc James (talk · contribs · email) 07:39, 28 November 2014 (UTC)
Doc James, your point on AHRQ is well-taken and we need to adjust wording for this. But otherwise it seems to me that you are campaigning for an election that was held last week. My read was that we had a solid majority for the idea that the consensus treatment guidelines are a better reflection of scientific consensus than meta analyses (whoever performs them), as these serve a different purpose. The controversy was mainly about whether to single out Cochrane reviews as especially deserving of mention when they were not aligned with consensus. While some people's viewpoint was not clear due to ambiguous language, I count only yourself, Zad, and MrBill3 as unambiguously opposing the proposal. Formerly 98 (talk) 09:15, 28 November 2014 (UTC)
Wording was poor. We could try a RfC to bring more eyes to the matter. It is not like that many people weighted in. I do not count more than a couple of people supportive. Doc James (talk · contribs · email) 09:31, 28 November 2014 (UTC)
James, I know you feel strongly about this but I think reverting the reversion of your reversion is taking us a little close to the E-cigarette model. I won't respond in kind with a fourth reversion, but I don't think this was ideal. We won't settle this by edit warring.
I don't think the question was poorly worded, but one problem was that we modified the wording of the edit in response to people's commentary as we went along, and people who participated in the early part of the conversation did not participate in the final part. That being said, I still don't see the validity of the main decision in doubt. I count 7 clearly in support who did not want Cochrane to receive special mention as a minority opinion, and 3 more who supported with a preference that Cochrane should routinely get mentionn as a key minority viewpoint when not in the mainstream. This out of 13 who expressed a discernable opinion. I'd welcome more voices, but am unsure what strategy will bring in more voices than what we originally did, which was to post this directly to the Medicine Talk page.
Another concern I have is that I believe that advance designation of Cochrane's opinions to recieve special attention in our articles violates WP:NPOV's requirements that we weight opinions according to their adherance among experts. The other organizations we cite are deliberately creating consensus documents, and so seem to me to be compatible with this requirement. WP:NPOV takes precedence over any decision we make here, but I'm not sure what an appeal would look like, as the NPOV board may have trouble understanding the issues.
What is the wording of your proposed Rfc? Formerly 98 (talk) 12:36, 28 November 2014 (UTC)
Yes I do not support the mention of Cochrane as a minority opinion because its isn't. It is a major opinion. We have been having this disagreement for a good year now. So obviously you aswell feel very strongly about it.
I presume that this pertains to psychiatric medications at least in part. The APA is the most financially conflicted of any medical association and they come out with very favorable opinions regarding medications. Less financially conflicted organizations come out with less favorable opinions. I consider this an attempt to reduce the position of one of the most highly respect and independent medical organizations globally so that only the position of the APA remains (or at least it is more prominent).
"according to their adherance among experts" is expert opinion which is the lowest level of evidence. Expert opinion should not be given greater weight than the evidence.
But let me ask you. Why do you think this change is needed? What issues do you see on Wikipedia that need this change to be addressed? I think that would help all of use consider things. Doc James (talk · contribs · email) 13:02, 28 November 2014 (UTC)
Apologies, family responsibilities. Back in an hour or two. Formerly 98 (talk) 13:29, 28 November 2014 (UTC)
  • Reviewing the concerns you raised above. Here is the Ezetimibe article before you edited it [4]. There was NO reviews stating that it should not be used. There was 4 primary sources commenting on "panels of experts". The only review in the lead was the NICE statement. Those primary sources should be removed and they were.
  • The controversial article that you hesitate to mention I assume is the e-cigs article. I assume that the article in question is the one in a no name journal with an impact factor of zero. Our guideline already deals well with this. We use reputable medical journals. This one wasn't.
  • So what we are left with is the Aripiprazole case no? Now if only half of people finish a 6 weeks RCT how strong of conclusions do you think one should be able to make from the resulting data? Doc James (talk · contribs · email) 13:36, 28 November 2014 (UTC)
You guys seem to be having personal/subject-specific issues with your sources/authorities which are crossing over into what you want to write in our general medical guidelines. This is no way to write guidelines because it just seems neither party is willing to actually read what the other guy says. Sorry, but "Cochrane" is not an opinion nor an authority. That way of viewing it is about as valid as James claiming to be the voice of Canadian healthcare. The papers I cited and criticised above are not the voice of a global medical organisation but merely that of a few doctors who happen to belong to it. In one example, those were doctors who have no experience of the subject. I think James you are trying to over-egg the name "Cochrane" to make it sound like one voice that must be heard. In comparison, a NICE guidelines is the result of a consensus of experts who weigh the evidence and bring in their experience where evidence is lacking. This combination is absolutely vital for there is only first-class evidence of very little. Nobody is suggesting we give weight to experts who disregard the evidence in favour of their own opinions, but neither does a organisation like Cochrane always produce the evidence you think is vital. Tell me, James, does that epilepsy paper provide the evidence that "phenytoin and carbamazepine appear to be equally effective in both focal and generalized seizures"? I suspect you don't understand that it doesn't or won't admit it is pretty useless as a source for anything, because that would detract from your argument here. Please take your topic-specific battles elsewhere. -- Colin°Talk 13:47, 28 November 2014 (UTC)


It seems a little unfair to me to suggest that I am raising this issue based on content disputes on a handful of pages. It has the effect of trivializing my position without actually addressing the points that I've raised. Its very effective as a rhetorical technique, but I'd like to think we can address each others concerns and points more directly than this based on the mutual respect we have developed working together.

James asks a very good question: "Why do you think this change is needed? What issues do you see on Wikipedia that need this change to be addressed?" Let me try to answer that.

  • I obviously spend a lot of time here. According to my wife, way too much time. I care about the quality of what we produce, else I would spend my time doing something else.
  • I agree that the Evidence Based Medicine movement has brought a new perspective to how we view a lot of these issues, and that it brings a lot of value
  • On the other hand, I do not see the technique of meta analysis as providing a formulaic method for discovering the "truth". It is not simply a mathematical distillation of the results of many trials. There are many, many subjective judgment calls involved in performing the analysis and writing the conclusions.
  • I think we all implicitly agree with my point above, in that no one trusts meta analyses performed by industry. Taking this as evidence that we believe the results of a meta analysis is subject to author bias, treating Cochrane meta analyses as the final arbiter of Truth requires we assume that Cochrane authors have no biases. That seems to me to be a difficult assumption.
  • My thought is that the best way to handle this and determine medical consensus is to do a little averaging. The opinion of 2 authors who chose to work together may not be representative of mainstream thought. The consensus opinion of 25 authors brought together to create a treatment guideline is more likely to average out the "tails" of the distribution curve, and give us a better estimate of the "middle of the road" opinion. In general, and not just in the specific cases I used for examples, I am troubled that the document that represents the opinion of 2 authors outweighs the guideline that outlines the negotiated joint position of 25 authors
  • James, you mentioned that 'according to their adherance among experts" is expert opinion which is the lowest level of evidence' But this is the standard in WP:NPOV, which takes precedence over decisions made at the project level. I think we need to be in compliance with Wikipedia's pillars, and if you have a problem with this standard it should be taken up at the level of NPOV policy.

With respect to Colin's comments, I'd be interested to know if he has a specific policy position that he is supporting here. I've seen a lot of fairly harsh criticism of both my comments and James, which has sometimes bordered on ridicule, but if you have offered a specific alternative, I somehow missed it. Criticism in the absence of offering a meaningful alternative doesn't really help us reach a consensus.

  • Doc James -- this reversion and the discussion above are pretty ugly. With that reversion, you took out content that everybody but you agreed to; and above you are personalizing the discussion. Both are out of line. Please continue to advocate for your position on cochrane, on the merits. If you can convince other folks, great, but if not, per WP:CONSENSUS, then you'll need to walk away. Thanks. Jytdog (talk) 05:43, 29 November 2014 (UTC)
I am seeing the poll below User:Jytdog and there is no consensus. Thus "everybody but you agreed to" is wrong. Yup agree this is getting fairly ugly.
The current summary of this page is "Ideal sources for biomedical material include general or systematic reviews in reliable, third-party, published secondary sources, such as reputable medical journals, widely recognised standard textbooks written by experts in a field, or medical guidelines and position statements from nationally or internationally recognised expert bodies." I assume that this proposal is trying to change that? Doc James (talk · contribs · email) 06:51, 29 November 2014 (UTC)

Another comment by a new reader of this thread:

Clinical guidelines from organizations the ultimate? For my two cents, I too agree that clinical guidelines from groups of experts are probably the best we’re going to be able to do as a simple algorithm for what is representative of NPOV in clinical medicine. You can secondarily mention the Cochrane Library and database and use these, of course, but remember that many clinical guideline formal recommendations use Cochrane data also, so you get that for free. For example, NICE has cited Cochrane many times as evidence which they used to formulate their guidelines. Not as much as Cochrane wishes, no doubt, but they do. [5]. So cite Cochrane particularly if your guidelines miss some nice Cochrane review you found (perhaps a later one), or it contradicts the guidelines. And while we are at it, we should remember that Cochrane is not the only source of clinical trial meta analysis! Sometimes something just doesn’t exist as a Cochrane meta analysis, like (say) the effect of fibrates on cardiovascular outcomes. No such review exists (and yes, that’s shocking, but illustrative). You CAN get a meta analysis that used the Cochrane library of fibrate trials, as well as MEDLINE and Embase, BUT using the library of trials kept by Cochrane does not make this a Cochrane review, with the Cochrane label. But this one is published in LANCET and is perfectly useable as a meta analysis. [6] The problem is knowing how to translate it into guidelines. NICE guidelines really don’t recommend fibrates for anything (maybe horrible primary triglyceridemia), but the meta analysis says they lower severe MI incidence by 10%, without impact on mortality. So now what? I’ll save that for later.

Cochrane not enough. Guidelines often utilize data from epidemiology and even animal and pathophysiology data, which Cochrane (as a strictly clinical trial evidence entity) does not. This is good and bad. The good is that it lets you include recommendations for having a parachute when you jump from a plane (very funny example above). The downside is that pathology and proxy marker data can fool clinicians into prescribing drugs that are no good (examples on request if you don’t have favorite ones of your own—mine is encainide). This problem is why some docs used niacin as a cardiac preventive up until mid-July, but there were also problems of underpowered studies with niacin, and other studies just contradicted each other.

Other meta analysis available. We need to remember that what Cochrane brings to the table is a sort of “guarantee” (warrantee?) that they don’t let chemists do reviews of surgery articles. I had more confidence in them before reading user:Colin's Cochrane complaints above, but what can you do? Again, product quality problems cannot be detected until you are a sophisticated user; sometimes just the GG Gucci label on something is not enough, because it get stamped on crap sometimes. You can use impact factors, but good journals print crap, too. And are biased. And print biased stuff. The routine male circumcision wars in JAMA a few years ago were hilarious, partly because each side took itself so medically and scientifically seriously, except the angry letters and reviews were always urologists Smith, Jones and Lee vs. urologists Cohen, Cantor and Mohel on the opposing view.

Primary studies often contain secondary material of high quality. When you have an article that is from a very large and long clinical study that is nearly definitive and is unlikely to be repeated soon, I think it’s a little silly not to simply go to that primary article’s “discussion” and “conclusion” sections for your secondary material (even primary science articles have secondary-discussion and synthesis parts, per the meanings of “secondary” in WP:RS). That’s another topic for discussion in MEDRS. It’s ridiculous to wait until an article in the NEJM is picked up and reviewed and compared to previous studies by the Lake Wobegon Family Practice Quarterly, when the authors in the NEJM almost surely did a better job in the original, biased though they may have been. And the editorial NEJM staff probably oversaw their conclusions with a more jaundiced eye than the Lake Wobegon editor, who was glad to just have enough material to make his page count.

Epistemology always bites when on the internet, nobody can tell you’re a dog. Ultimately we came back to the epistemological question I complained of several years ago, which is that you cannot write a good NPOV science article without appeal to authority, and that appeal rests on quicksand at some point, since (without ability to do a background check of any type) you very quickly run out of authoritative views on who is an authority. And if you make it that far, you won’t be able to find authorities on who is the authority on finding authorities. At some point, you cannot avoid just taking polls of the schmucks who post here. From our mother’s basements. Ultimately, it’s impossible to write a technical WP article with a reasonable NPOV without breaking guidelines on whether or not you should trust certain editors who write the things. You watch to see if the editor fixing the Lie Algebras article looks like their LaTeX is up to snuff (or broken and ugly), and you sense from their TALK input whether or not they know what they are talking about, mathematically. Something cannot be done by cites, or cites of cites, or cites of meta reviews of reviews of primary cites. The other way (editor trust) is actually how most good articles on WP get written, though we deny it. So I’m here (as usual) to remind you all of the fundamental hypocrisy that is Wikipedia. If Wikipedia were actually written the way that it is supposed to be (no recognizable editors involved—we randomize all usernames with each edit, so there are no possible reputations), Wikipedia would be a lot crappier than it is. However, we use longstanding usernames, and we get to know them (and can look up their contribution histories). And that works as a better proxy than we’d like to admit, for the editor CVs we lack. SBHarris 00:57, 26 November 2014 (UTC)

====Re-poll==== OK, I think that @Yobol: has just backed us up to a version that is less supported than the version that we had before his edit, so I'm going to do a poll. Please let me know if the following is roughly correct:

I have the following people having expressed the opinions below at some point. Let me know if this is correct or not:

Mostly agree that treatment guidelines are an "ideal" source for determining Scientific Consensus, and by implication, should get more weight (in the sense outlined in WP:NPOV) than other secondary sources, which are not consensus documents. As near as I can tell, this group opposes calling out Cochrane as a special source that should always be called out alongside the consensus treatment guidelines.

  1. @Formerly 98
  2. @Mastcell:
  3. @Colin:
  4. @Jytdog:
  5. @WhatamIdoing:
  6. @Yobol:

The following appear to feel that Cochrane should still get mention on an least an equal footing with consensus treatment guidelines, but seem to support emphasizing these (Consensus guidelines and Cochrane) over other secondary sources for determining Scientific Consensus:

  1. @Doc James: (or possibly disagrees with this proposal altogether, preferring the previous status quo/ Unclear)
  2. @Casliber:
  3. @Anthonyhcole:
  4. @Zad68:
  5. @MrBill3:
  6. @JFW:

Opinion unclear or difficult to categorize:

  1. @TenOfAllTrades:
  2. @LeadSongDog:
  3. @109.157.83.50:

Is this a correct count? I assumed that we were in agreement that the garden variety meta analyses and reviews should be weighted a little less than consensus treatment guidelines for determining Scientific Consensus, and were split on Cochrane. So I tried to split the difference on Cochrane, calling them an important minority voice when not in agreement with consensus treatment guidelines. Let me know what you all think. Formerly 98 (talk) 02:58, 25 November 2014 (UTC)

Please don't do this. We discuss and come to a consensus. We don't vote or tally opinions. If you believe "a slight plurality felt that Cochrane's views always deserved mention even when they were out of synch" then I couldn't really care if a slight plurality felt the moon was made of cheese. They would be wrong. Some policy reading required. -- Colin°Talk 12:55, 25 November 2014 (UTC)

Do you have a specific suggestion on how to proceed then? We wrote into MEDRS what we percieved to be a reasonable compromise among the various viewpoints that were discussed here, and that compromise was reverted, so I really don't know how to proceed here. I'm not seeing a specific suggestion in your comment other than that we should write your own opinion into the guideline, and there is no consensus to do that. Formerly 98 (talk) 13:51, 25 November 2014 (UTC)
Somehow I missed the new version by Yobol, which seems fine to me. Formerly 98 (talk) 14:56, 25 November 2014 (UTC)
Still says "Wikipedia policies on the neutral point of view and not using original research...", which is still incorrect. How about "Wikipedia policies on the neutral point of view and not publishing original research..." instead? LeadSongDog come howl! 19:13, 25 November 2014 (UTC)
To avoid muddying the waters, I'll defer the above issue until the rest is sorted out.LeadSongDog come howl! 20:34, 25 November 2014 (UTC)
I have no problem with a poll to figure who is standing where. I guess my position is similar to Doc James, but as far as I can see, the wording reverted to here implies that Cochrane can be used (?) but is written in a weird double negative. Doctors consider it a secondary source equal to a review article. Formerly 98 yes I suppose researchers could have all sorts of vested interests etc. but on the whole there is (superficially anyway) more of a stance of neutrality. Fact is, economic rationalism everywhere means that about 98% of medical clinicians are being micromanaged to maximise clinical output at the expense of research. Hence there is now a higher proportion of research funded by pharma. I don't say Cochrane is unique, but is held on a par with various government and professional treatment guidelines. I consider the newer version an improvement BUT would prefer the inclusion of Cochrane in examples after WHO like it was before. Cas Liber (talk · contribs) 06:38, 29 November 2014 (UTC)
Agree Cas, the wording along the lines of "don't forget not to ignore..." really framed Cochrane in a negative way. Zad68 03:10, 30 November 2014 (UTC)

RfC on adding the following wording

There is clearly no consensus to make the proposed change. Number 57 13:38, 23 January 2015 (UTC)

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

User:Jytdog has proposed changing:

to

Oppose

  • Oppose While I do not believe the original wording is perfect I do not believe the new wording is an improvement. Many organizations produce excellent meta-analysis including: major medical organizations, scientific societies, AHRQ, USPTF and the Cochrane collaboration. It is unclear why we are separating out Cochrane reviews from other high quality reviews? Also the balance between practice guidelines and systematic reviews depends on the topic and specific issue in question. Some practice guidelines are good and others are less good and the same applies to reviews. Factors such as the target patient population they are geared towards and the up to dateness must also be taken into account. Cochrane and WHO publish material that is often more global in scope while most guidelines out of the USA and UK are very USA and UK centric. A blanket statement for one over the other is not a good idea as these issues should be addressed on a case by case bases. Currently we balance the positions from major reviews with those of the position statements of national and internationally recognized organizations and we should continue to do so. Doc James (talk · contribs · email) 07:14, 29 November 2014 (UTC)
  • Oppose. The new wording is problematic and too detailed to follow IMO. We can't have a broad statement to assert one review is much better than another. QuackGuru (talk) 07:22, 29 November 2014 (UTC)
    Your response surprises me, given the number of times you've said exactly that (e.g., that any review written by Edzard Ernst is much better than any review written by any chiropractor) in disputes. WhatamIdoing (talk) 10:51, 4 December 2014 (UTC)
  • Oppose as well, this wording appears to single out Cochrane as a minority opinion organization. I don't know if that was the intent but that's how it reads. If we're going to name names of organizations that we look to for the best-quality evidence, Cochrane needs to be in there along-side them. Zad68 02:50, 30 November 2014 (UTC)
  • Oppose Cochrane reviews are not the views of an organization: it's a professional society with two for-profit businesses, publishing a commercial journal, which is produced by 50 sub-groups. A Cochrane review is the opinion of its authors, published by one of those 50 sub-groups, not endorsed by the editors or any central organization. In addition, there is no reasonably unbiased source to show that currently, Cochrane reviews are better than systematic reviews in other journals, or those published by AHRQ and other similar organizations, many of which go to far greater lengths to ensure independence and avoidance of conflicts of interests than Cochrane groups do. Given that more than half of Cochrane reviews have not been updated for years - including many which have not had an update since the 1990s - they cannot reasonably be assumed to be superior to a recent review. The discussion also raised several examples of controversial Cochrane reviews. When there's a controversy, this is often even more likely to be the case: there is considerable campaigning via systematic review, with people defending positions across time and having developed an intellectual conflict of interest. Cochrane policies in relation to matters relevant to NPOV vary from group to group, and many issues relevant to NPOV are not dealt with better by Cochrane than others (e.g. a Cochrane review often includes the authors of the trials being reviewed, and this is not often not made clear to readers). Hildabast (talk) 17:13, 30 November 2014 (UTC)
  • Oppose, essentially agree with Doc James (talk · contribs) and QuackGuru (talk · contribs), above. Wording change as proposed seems to introduce potential problems rather than simple solutions. — Cirt (talk) 05:10, 1 December 2014 (UTC)
  • Oppose. Although I appreciate the intent of Formerly 98, I think this wording goes too far. My reasoning is pretty much aligned with Colin and Doc James, but I also think a good deal of what the intent is here is already covered by WP:RS in general. It's always troublesome when we give preference to any publication, organization, whatever. I'm reminded that the New England Journal of Medicine once printed (and never retracted) an incorrect definition of Tourette syndrome, and that the NIH Fact Sheet on Tourette syndrome for years (maybe still) had inaccuracies about the condition. All of these issues should be left to editor discussion regarding individual sources for specific articles. If we "prescribe" certain sources, someone will misuse them.

    Meanwhile, why is this still in the guideline? SandyGeorgia (Talk) 16:04, 5 December 2014 (UTC)

  • Oppose New isnt always better, organizations make mistakes, placing faith in one source over another can lead to subjective edits. AlbinoFerret 16:30, 5 December 2014 (UTC)
  • Oppose I love Wikipedia and Wikipedians. The grassroots way that we write policy and guidelines is wonderful! I'm commenting here as a long time member of the community who has taken an interest in science and medical topics, and who is currently a Wikipedian in Residence at Cochrane Collaboration. I have a problem with the proposed wording for several reasons. 1) The new wording overemphasizes one type of source (guidelines and position statements) when we really need for people who are writing Wikipedia health content to be using up-to-date quality sources of all types. I want to encourage people (especially students) to use up-to-date textbooks which usually give a much better overview of medical conditions and treatments than guidelines or systematic reviews. The next step is to add more current information which which can come from a variety of reputable sources. The average person looking for advice about which source to use should feel okay about up-to-date systematic reviews/meta-analysis research as well as up-to-date clinical guidelines from established and well respected groups that disseminate health information. I strongly want to encourage the use of all of these types of publications because no one type is providing the best coverage in all medical topic areas. Decisions need to be made on individual articles considering the available sources. 2) The new wording is misleading because it gives the impression that there is "medical consensus" from these prominent organizations that establishes universal methods of diagnosis and treatment. It is wrong for Wikipedia guidelines to mislead the less informed writer by giving the impression that adding content from prominent organizations guidelines is stating the most common or consensus method of diagnosis or treatment. I suggest that we tone down the wording to eliminate that impression. 3) Wikipedia guidelines needs to reflect that the consumers of health information expect up-to-date information. Wikipedia itself has proven that the publishing world is forever changed, and people are seeking out the latest reliable information online. Both consumers and providers of health care are looking for ways to find the most reliable up to date information. We are doing a huge disservice to our readers if we overemphasis committee written statements (which will always be outdated) over systematic reviews/meta-analysis which could be getting better information into the hands of providers and consumers. 5) Our readers and writers are from all over the world. To overemphasis a regional organization's guidelines at the exclusion of systematic reviews/meta analysis makes our content less global. Page view stats It has the potential to introduce biases in content that might not be well understood by people outside of a particular region. 6) I want to strongly re-iterate that Wikipedia articles need a variety of types of sources in order to be properly written. I fear that the needs of the average writer and reader is being overlooked because some of our current medical editors are grappling with how to deal with a particular type of editing disputes. I strongly encourage people to engage in dispute resolution on individual articles and not solve their disagreements by re-writing policy in a way that could have detrimental unintended consequences. Sydney Poore/FloNight♥♥♥♥ 19:04, 5 December 2014 (UTC)

Support

  • Support WP:NPOV strictly requires that we "Ensure that the reporting of different views on a subject adequately reflects the relative levels of support for those views". As I have argued above, consensus treatment guidelines are designed to reflect consensus, and thus are an excellent guide to what the Scientific Consensus is. Meta analyses serve an important but different purpose. They often reflect positions that are significantly outside the mainstream. Formerly 98 (talk) 11:51, 29 November 2014 (UTC)
  • Support Wikipedia presents the mainstream views - the consensus - in all fields, with appropriate weight. This is the essence of NPOV. Recent treatment guidelines are by far out best sources for medicine. Science-based medicine is important and Cochrane collaboration's mission to drive SBM is great. But we are not Cochrane. WIkipedia is WP:NOT on the cutting edge of anything, and we are not out to SOAPBOX or try to drive change in any field. Where a science-based recommendation has entered the mainstream and driven treatment guidelines, great - that is what we transmit. But otherwise, leading with Cochrane is just WP:RIGHTGREATWRONGS. We need to be behind the curve, not in front of it. The change that Formerly 98 suggested is an important corrective. Additional note. I am not married to the exact language I implemented. What I objected to, was Doc James putting CC reviews on par with statements by major medical and scientific bodies as he did in this dif. ;the Cochrane Collaboration is great and does important work, but it is a loose collective that produces uneven results. Jytdog (talk) 02:52, 30 November 2014 (UTC)
  • Support This seems like a more clear explication and provides guidance on the balance between consensus and good sources (Cochrane) that provide important considerations/data. I'd think major institutional reviews deserve inclusion but in a balanced way. The clear emphasis on where the extant consensus can be found is useful. - - MrBill3 (talk) 03:13, 30 November 2014 (UTC)
  • Support I agree that WP shouldn't champion what's "correct" over what's stated in mainstream WP:RSs. If the proposed wording helps us do that, fine. The wording is longer than I'd like, but I guess compromises generally look like this. --A D Monroe III (talk) 18:55, 3 December 2014 (UTC)
  • I think I can support this change, except for the last sentence. I'm not happy with the way Cochrane is being presented, and I would omit it and leave that discussion for another day. Jytdog's statement about Cochrane's "uneven results" resonates with me. WhatamIdoing (talk) 10:54, 4 December 2014 (UTC)

Discussion

This proposal was initiated by my concern that meta analyses are being inappropriately used to indicate Scientific Consensus, when that is not their purpose. They play an important role, but due to the large number of subjective decisions made in their preparation and interpretation, invariably reflect the pre-existing biases of the authors. Per published meta analyses:

Treatment guidelines consider and integrate all these results, help provide expert analysis of the conflicting results, and reduce cherry picking. Formerly 98 (talk) 12:58, 29 November 2014 (UTC)

So with respect to varenicline than we should go with the FDA from 2011 [7] or the AHSF [8] that it might cause cardiovascular disease? Or should we mention the Circulation meta-analysis that says it doesn't [9]. I am of the position that the more recent meta analysis in Circulation should get prominence as the CMAJ and FDA are from 2011. The bigger issues is the use of primary sources in that section.
With respect to thimersol, the one review is from 2006 and an unknown journal. The other review is from 2014 in vaccine. It is not hard to say which we should go with.
Statins for primary prevention are controversial both within the meta analysis and the guidelines. More supportive evidence has recently become available.
The problem with treatment guidelines is that they may be old and they may not cover the issue in question. The AHA guidelines for cardiac arrest are only published every 5 years. We need to be a little more updated than that. Doc James (talk · contribs · email) 13:52, 29 November 2014 (UTC)
Agree on the need to be up to date. Generally I avoid using treatment guidelines that are more than 2 years old. I picked poorly on the varenicline meta analyses example, but there was a second meta analysis published in BMJ that contradicted the CJC meta analysis within 1 year. I believe it covered the exact same data set. Formerly 98 (talk) 14:08, 29 November 2014 (UTC)
Guidelines come to different conclusions all the time. Check out the initial choice of medications recommended for hypertension. "The Cochrane collaboration, World Health Organization and the United States guidelines supports low dose thiazide-based diuretic as first line treatment.[71][84][85][86] The UK guidelines emphasise calcium channel blockers (CCB) in preference for people over the age of 55 years or if of African or Caribbean family origin, with angiotensin converting enzyme inhibitors (ACE-I) used first line for younger people.[87] In Japan starting with any one of six classes of medications including: CCB, ACEI/ARB, thiazide diuretics, beta-blockers, and alpha-blockers is deemed reasonable, while in Canada and Europe all of these but alpha-blockers are recommended as options.[59][84]"[10]
How about guidelines for the use of chiropractic treatments for back pain? "National guidelines reach different conclusions, with some not recommending spinal manipulation, some describing manipulation as optional, and others recommending a short course for those who do not improve with other treatments.[17]"
Good guidelines are based on meta-analysis. Shifting our focus more to guidelines may in some cases simply make us less accurate as we will be more out of date. Doc James (talk · contribs · email) 14:16, 29 November 2014 (UTC)
I agree with you that the guidelines are not completely consistent. But overall, if you look at the various treatment guidelines for depression, schizophrenia, osteoporosis, and the like, I think you'll find more consistency than in the meta analyses, at least that has been my experience. I attribute this to signal averaging. A meta analysis may have as few as a single author, where as the treatment guidelines represent a compromise document hammered together by a much larger group. That is my concern, that we are placing more weight in some cases on the opinion of a handful of authors than on consensus documents put together by larger groups. And as the Cochrane example above shows, there is not necessarily any linear narrowing of opinion over time in meta analyses as more data becomes available. Sometimes they just change their mind. Its not always a more informed analysis, just one influenced by different or changing opinion. I sense your strong opposition to what I am proposing here, my question would be do you have a proposal that addresses these concerns but which does not raise your hackles like this one does? Formerly 98 (talk) 14:25, 29 November 2014 (UTC)
I do not see significant issues with what we had before. I think each of the examples you mention needs to be discussed on a case by case basis and can be solved or at least improved that way. If we are looking at historical issues like mercury in vaccines one must also remember that a guideline declared homosexuality a mental illness. Guideline committees are not random groups of experts but often very carefully selected ones which occasionally all share a similar POV. I am specifically thinking about Zamboni and CCSVI as one example. Would we want guidelines from these organizations [11] to be given any weight? Doc James (talk · contribs · email) 14:29, 29 November 2014 (UTC)
Yes, I realize you did not see a problem with the old guideline, but some of us did. My question was directed to trying to come up with something that was mutually acceptable and not simply having this end up as a binary vote in which one side loses. That would not be best if there is a creative compromise out there somewhere. Formerly 98 (talk) 14:41, 29 November 2014 (UTC)
Sure so my position is that when conclusions of various high quality sources disagree we should simply state each conclusion with attribution. We do this for hypertension, chiropractic and breast cancer screening. With respect to the latter at various times we have had people try to delete the position statements and at others the meta analysis depending on what POV the come to the article with. Doc James (talk · contribs · email) 14:54, 29 November 2014 (UTC)
  • I don't find voting helpful, and is generally discouraged on wikipedia. I think the first two sentences of the proposed text are fine. The second two are either redundant or at odds with policy. The robotic, academic systematic reviews produced by Cochrane have their place on Wikipedia but that place is more limited than many supporters should admit. Formerly 98 is absolutely right that a systematic review produced by an algorithm is absolutely not any measure of consensus among expert authors, which is exactly what policy requires us to use as sources. James is concerned we may become out of date by emphasising guidelines -- well Wikipedia has no requirement to be bang up to date, otherwise we would employ experts to cite the latest primary research or newspaper journalists to rave about the latest cause/cure for autism/cancer. It seems to me that some here are interested in finding the best sources that meet our policy (what do experts believe, based on the best research) whereas others want only to report the very latest scientific analysis. This conflict between sources and Truth has always been a problem on Wikipedia and "the latest Truth" has always lost. We are a tertiary source. If the doctors treating us can manage to wait for the next set of guidelines to be published, I fail to see why a general encyclopaedia needs to be more up-to-date. We are documenting reality here, are we not, rather than what we would wish were the reality? I suggest if we cannot come to a general wording that pleases both camps that we just retain the old text. I often find citing WP:WEIGHT better than MEDRS. And remember, outside of the battleground of one drug vs another for very specific situations, the guidelines will offer a far richer basis on which to build a treatment section than the very narrow focus of a systematic review. -- Colin°Talk 16:22, 29 November 2014 (UTC)
It's not either/or

I think it's a false choice, to say that we have to go with or prioritize a practice guideline against high-quality (sorry, Colin, yes I know there are a few exceptions to every rule), independent source of evidence like Cochrane. As mentioned above, practice guidelines can recommend a course of action in cases where the evidence is less than excellent, and individual organizations can be subject to political pressure or other biases that an independent reviewer might not be subject to. So, why wouldn't we use both? The article can say, "The evidence says this. NICE recommends that." if that's the case. If we're going to name names of organizations that produce practice guidelines, we should also names well-respected, influential sources of evidence. Can anyone argue Cochrane is any less well-regarded than the named bodies producing practice guidelines? Zad68 03:05, 30 November 2014 (UTC)

  • Zad68 about your !vote above.... to me, this is not about evidence, but about presenting the mainstream medical views. (these are two different things...) I said this before, but in my view, the publication of any given CC meta-analysis is a WP:PRIMARY source for the results of that analysis - they sometimes end up changing clinical practice (a good thing!), they are sometimes rejected, and sometimes they affirm what was already done (also a good thing). The key question is - how do we know? The question applies when a CC Review first comes out, and when it has been around for a while. In my view, we know how any particular meta-analysis has been used (or not), by looking at treatment guidelines. In my view, we need to be wary of becoming evangelists for science-based medicine in general and for CC's products in particular. We should wait and see what the relevant field does with them. In my view, we should not pounce and push them into WP when they are first produced.... I am interested in your thoughts on this way of looking at them. Jytdog (talk) 03:05, 30 November 2014 (UTC)
    • Jytdog I may have to pick this up later, but earlier you wrote "The meta-analyses they produce are of course great secondary sources." Is that not your position any more? Everything is a primary source for itself, of course. We use the meta-analyses they produce though as secondaries regarding the evidence. Zad68 03:16, 30 November 2014 (UTC)
      • Zad68 Thanks for replying! My reply has two levels - one of which I am very clear on, and the other... where i am struggling. First and ver clear to me - meta-analyses are attempt to determine what the evidence is for treatment X or Y, based on clinical trials. Evidence is different consensus. (the science-based medicine movement and many of us wish that all treatment was based on evidence.... but as we all know the evidence is not always strong, and sometimes even doesn't exist, for choosing X vs Y). Presenting to readers something like "the evidence for X is weak" more prominently that "doing X", when doing X is the consensus, can be downright harmful, as Formerly has pointed out in some of the psych med articles. Cases like that, are places where Wikipedia has become an evangelist for evidence-based medicine as opposed to presenting the consensus. Secondly - and here is where my thoughts are in flux: as both Colin and Formerly have pointed out, there is a lot of subjectivity in the choosing of what trials to include or exclude, and in performing the analysis... and that really has pointed up to me that we need to treat meta-analyses with great care... they are a weird animal, with much akin to WP:PRIMARY sources. It is not clear to me how we should classify them... but it is clear to me that we should treat them with care. Thanks again for replying. Jytdog (talk) 16:02, 30 November 2014 (UTC)
        • Jytdog I don't disagree with anything you're saying here but I still think there are two things here we need to be careful not to confuse -- There are reviews of evidence, and there are practice guidelines. From what I've read of them, Cochrane reviews are careful to avoid making a recommendation in their conclusions. They may say "Evidence supports doing X," or doesn't, or that evidence is insufficient to draw a conclusion (probably a MAJORITY of Cochrane reviews say that), but I haven't seen them say "Therefore patients with such-and-such a condition should do X." That's what practice guidelines do. Practice guidelines can make recommendations where evidence is missing or unclear (c.f. the parachute problem elsewhere in these threads), or practice guidelines can even make recommendations against what the evidence might indicate, due to other, real-world factors. I don't see these as in conflict, I see them as complementary.

          Am I meeting your point or still talking past? Zad68 19:52, 30 November 2014 (UTC)

          • Zad68 thanks for replying and I should have replied directly sooner. I do hear you on the not either/or thing. Generally I think you are right on with that. The issue is what to do when the guidelines and Cochrane conflict. I didn't think this used to be a big deal but it is a problem (for example) for pretty much the whole class of psychiatric drugs, for which the data is generally pretty poor and we have few good options for people with these diseases... for a whole host of reasons that I am sure you know. A lot of this emerged in efforts by Formerly 98 and Doc James to come to agreement on how to handle Cochrane reviews of antipsychotic drugs. See for example the edit history of Aripiprazole and its Talk page, specifically the Talk section Talk:Aripiprazole#Cochrane_reviews_of_antipsychotics (and the table that Formerly built at the bottom of that section, showing how Cochrane reviews come to pretty different conclusions based on similarly messy data). If you have some time and can really look over that, you will see what I mean. In those discussions Doc James has a very clear position and Formerly is working his way into his, so it is a bit messy. But I've decided that the place where Formerly was struggling toward is correct (namely, that conclusions of meta-analyses can be fickle and we should treat them carefully and perhaps somewhere between PRIMARY and SECONDARY) and that Doc James was not. Since then I have been thinking it through more and more, and seeing the larger pattern here of slipping into advocating for EBM. Which is what prompted my opening a section directly on "consensus vs evidence" below. I hope that is more clear. Jytdog (talk) 15:18, 1 December 2014 (UTC)
  • I am happy with Yobol changes [12]. We already have enough issues encouraging the wider community to use reviews from high quality sources, practice guidelines or major textbooks rather than primary sources and the popular press. Making our list of suitable sources even more restrictive I have a feeling will not go over well. We do not simply get to decide that CC review are "primary sources" when the rest of the scientific and medical community disagrees with this.[13][14][15] Doc James (talk · contribs · email) 04:15, 30 November 2014 (UTC)
  • I think that part of what has been raised here is the fact that a systematic review/meta-analysis is an interpretive work, of 'observational' rather than 'experimental' design. One of the problems emerging is that of what to do when there are conflicting positions evident in key secondary sources - whatever they are. That can't be sorted out on the basis of who produced it. It would make life lots easier if it were true, but it's not. I wonder, having tried to read and absorb this thread just now, if whether what would help is some guidance on common things to look for, in areas where the secondary sources disagree, and then how to present that? Hildabast (talk) 17:26, 30 November 2014 (UTC)
I'm not sure I'd call 2 college libraries and a blog "the rest of the scientific community". But in any case I don't think the argument hinges on primary vs. secondary. The standard set in WP:NPOV is to "Ensure that the reporting of different views on a subject adequately reflects the relative levels of support for those views". While this debate was ongoing, you added to the lede of the aripiprazole article a Cochrane statement "Evidence for its effectiveness is not that strong" in Wikipedia's voice, in spite of the fact that it is recommended on at least an equal footing with other antipsychotics by every schizophrenia treatment guideline that I am aware of, at least one of which calls evidence for its effectiveness "Grade A". I find that decision very difficult to square with WP:NPOV's insistence that we describe viewpoints "in proportion to their prominence". WP:NPOV simply does not allow us to elevate our favorite EBM source over more widely held viewpoints. It is one of Wikipedia's pillars, and as such "is nonnegotiable, and the principles upon which it is based cannot be superseded by other policies or guidelines, or by editor consensus". It does allow, and in fact require, that all signficant views be mentioned, but in accord with their prominence. Formerly 98 (talk) 09:56, 30 November 2014 (UTC)
Going a bit off-topic here, but aripiprazole is found in some review articles to be slightly less effective than other antipsychotics - however, it has advantages in the lack of weight gain and prolactin serum level rises associated with it, so Cochrane isn't exactly contra. Cas Liber (talk · contribs)
Fair point, drifting off topic, and struck. Good observation, Casliber. Formerly 98 (talk) 10:16, 30 November 2014 (UTC)
  • I think Cochrane should get specific mention, even when it contradicts established consensus. Even the best secondary sources sometimes contradict each other, and I would not want to gloss over a Cochrane review just because it is not in line with established practice (which may be based on poor evidence or even expert consensus). Cochrane gets cited very widely as a definitive (or at least authoritative) source. JFW | T@lk 09:43, 30 November 2014 (UTC
I think the proposal as written encourages exactly that JFW. Formerly 98 (talk) 09:56, 30 November 2014 (UTC)
It appears that the question is how should we balance EBM and expert opinion. Some feel one should get greater weight over the other and vise versa. My position is they should often have equal weight or that at least weight should be determined on a case by case basis. I do not believe that this is something for which we should have a blanket statement in this guideline. There is already sufficient attempt to guide people in what sources they should use. I have people complaining about not being able to use primary sources on a daily bases. Doc James (talk · contribs · email) 10:13, 30 November 2014 (UTC)
EBM always trumps expert opinion. If both sources disagree, we put more weight on statistical evidence and less on personal opinions. -A1candidate (talk) 14:34, 30 November 2014 (UTC)
It may do so in many circles A1, but not in Wikipedia's NPOV policy. The term "evidence based" does not appear anywhere in the document. The specific language of WP:NPOV is
  • "Giving due weight and avoiding giving undue weight mean that articles should not give minority views or aspects as much of, or as detailed, a description as more widely held views"
  • "in determining proper weight, we consider a viewpoint's prevalence in reliable sources, not its prevalence among Wikipedia editors or the general public."
  • "Neutrality assigns weight to viewpoints in proportion to their prominence"
and finally,
  • "This policy is nonnegotiable, and the principles upon which it is based cannot be superseded by other policies or guidelines, or by editor consensus."
To me, this language is unambiguous. We cannot attribute one source as the "gold standard" and routinely give it precedence over others when sources disagree. And which statistical analysis would you accept in any case? The one that says Prozac is worthless or the one that says it is efficacious? There is no pure form of "Evidence Based Medicine". Opinion is everywhere, in the decision about which trials to include, which endpoints are worth evaluating, in the decision as to what issues to emphasize in the conclusions.
Formerly 98 (talk) 16:03, 30 November 2014 (UTC)
The statistical method to be used depends on the clinical question being answered. With regards to SSRIs and many other medications, a meta-analysis of effect size might be most suitable for determining efficacy. If several systematic reviews come to different conclusions, we could look for what is called a systematic review of systematic reviews. -A1candidate (talk) 16:49, 30 November 2014 (UTC)
Doc James wrote, It appears that the question is how should we balance EBM and expert opinion.--Yes, exactly how I see it, and: Some feel one should get greater weight over the other and vise versa. My position is they should often have equal weight or that at least weight should be determined on a case by case basis.--Yes, exactly. I still think there's an incorrect assumption underlying some of the comments in this discussion that it's either one or the other, or that one has to be prioritized over the other. I really see a top-tier source of evidence like Cochrane and a top-tier authority's practice guidelines as right there next to each other, neither should be prescriptively prioritized over the other, and the use of each, including the weight, needs to be handled on a situation-by-situation basis.

Sbharris wrote (above) a really awesome, laying-the-truth-bare mini-essay on how the sausage really gets made here, and I'm disappointed it really didn't get any traction. Zad68 19:20, 30 November 2014 (UTC)

Guidelines and systematic reviews are different things typically. Guidelines often address a much broader question while systematic reviews address a much narrower one. Thus while guidelines often have more authors all these authors many not be looking extensively at each question. Thus it may be less authors than a systematic review. Not that I am suggesting we determine weight by how many authors a review or guideline has. Doc James (talk · contribs · email) 03:11, 1 December 2014 (UTC)

Moving review articles from "ideal source" to "other source" is an issue. They are all three potentially ideal sources (review articles, position statements/policies, and major textbooks) Doc James (talk · contribs · email) 15:48, 5 December 2014 (UTC)
The discussion above is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

A specific example

The wiki ganfyd states here that activated protein C is grade A recommendation for pneumonia based on a 2008 guideline from British Society for Antimicrobial Chemotherapy. The ref is here [16]. However a 2011 Cochrane review found that it was associated with worse outcomes.[17] It took the FDA 6 months to pull the drug at which time UpToDate updated their recommendations. Wikipedia reflected the best available evidence within a couple of days of it coming out. Doc James (talk · contribs · email) 12:44, 30 November 2014 (UTC)

Correct me if I'm wrong, but it appears that APC was given recommendation grade B only. -A1candidate (talk) 15:16, 30 November 2014 (UTC)
Yes they misinterpreted the guideline plus the guideline is old. Doc James (talk · contribs · email) 03:00, 1 December 2014 (UTC)
I'm very skeptical that the FDA is actually able to move that fast. Just kidding. But I'm not sure that Wikipedia should have moved that fast. We present scientific consensus. By changing our content on the basis of a single source when others did not yet reflect a change in opinion, aren't we getting ahead of what a tertiary source should do? We haven't updated Ezetimibe on the basis of IMPROVE-IT yet, and will not do so until the result is confirmed by other sources. Updating in both cases violates Wikipedia principles. In the case of ezetimibe, that of working from primary sources, in the case of APC, that of not giving viewpoints weight according to what is published in reliable sources. Formerly 98 (talk) 16:12, 30 November 2014 (UTC)
In the ezetimibe case we are not using a PRIMARY source. While in this case we are supporting using a SECONDARY source.
Guidelines are often TERTIARY sources. We should not try to becoming a quaternary source based only on tertiary sources. We should be based on both tertiary and secondary sources.
Doc James (talk · contribs · email) 03:00, 1 December 2014 (UTC)
There's no such thing as a quaternary source. Some academic fields (including law) do not even have tertiary sources. WhatamIdoing (talk) 12:44, 1 December 2014 (UTC)
Yes I know :-) And we should keep it that way. Doc James (talk · contribs · email) 05:28, 2 December 2014 (UTC)

In addition to the embarrassing nonsense about quaternary sources, the above cherry-picked example could (a) be repeated (with other studies/papers) as an example of why Wikipedia should cite primary research papers rather than wait around for reviews or textbooks to pick up on the latest findings and (b) merely shows that new publications tend to include more up-to-date information, which is a tautology. We are looking for sources that reliably indicate medical scientific consensus on a topic, because we are an encyclopaedia not a newspaper. We are limited by the speed at which reliable publications pick up and report on this consensus. If we were written by named expert authorities in the subject, then we could be both up-to-date and reliably cite the latest research findings. We aren't and we don't. -- 11:46, 7 December 2014 (UTC)

Source kinds and EMB vs Expert opinion

The primary/secondary sourcing terminology for science differs to that for humanities. The first publication of new research based on previously unpublished data is a primary source. So if (as in the Epilepsy paper used above) a Cochrane review decides to escape the confines of analysing previously published results and instead sources the raw unpublished patient records upon which to perform novel analysis, then this is a primary source. In addition, "all sources are primary for something". And since science always builds on the work of others, and this work is frequently referred-to in a research paper, all primary research papers will include some secondary-source material. It really depends what you take from the paper and how you use it on Wikipedia. So, in the words of a notable EBM advocate, "I Think You'll Find It's a Bit More Complicated Than That" -- don't oversimplify the argument.

We all know that if anyone is likely to produce a "There isn't enough high quality data available" result in their publication it is a Cochrane review. That's why sceptic bloggers battling against pseudo science and snake oil merchants love to cite them. But back in the hospital ward, the physician needs to know what drug to give a child for a certain problem but who additionally has a serious metabolic condition, even though the only EBM randomised controlled trial is on young otherwise-healthy men. Nobody conducts randomised controlled trials on pregnant women to see which drugs cause the most defects in their babies. We have an RCT on refractory epilepsy treated with the ketogenic diet but absolutely no RCT studies on refractory epilepsy (AFAIK, PMID 22170887) treated with drugs (after all, what manufacturer would choose that patient group). Yet the physician, faced with a child who has failed three drugs already, must choose a fourth or some other treatment. To claim, as proposed below, that several RCTs combined in systematic reviews is required before making any guidance or reporting something on WP, is just on another planet.

The old chestnut of "expert opinion" vs "evidence based medicine" is repeated and misused. When these are compared like this, the implication is that the "opinion" is "mere opinion", i.e., not based on good evidence, but rather on either prejudice, limited personal experience or plain old guesswork. If Wikipedia was based only on evidence, then we'd cite nothing but primary research papers and systematic reviews and discard literature reviews, textbooks, clinical guidelines and consensus statements. But then we'd need to be authored by respected named experts who could interpret that evidence properly. Because the evidence is not as general as we need for WP article. We have evidence that of a 200 patients treated in a certain trial by a certain hospital according to a certain protocol and given a certain dose of a drug, that 100 got better. And our science may tell us to what confidence we can then attribute that to the drug. And we can gather data on the side effects seen in this short-term trial. Is "half the patients getting better" an effective drug? Depends on the condition. I wouldn't be happy if my optician only prescribed glasses that worked half the time. On the other hand, some cancers never respond anything like that well. If one in a hundred of those patients got liver failure and died, that might be acceptable for fatal cancer but not for irritable bowel syndrome. The decision about what makes a good drug and therefore worthy of mention in the "treatment" or "management" section of our disease articles, is one of expert opinion. So it always ultimately comes down to the opinion of experts and what they have written in reliable publications. We trust that the best experts and best publications will ensure that opinion is based as much on high-quality evidence as possible, but it is not Wikipedia's job to fix that if we disagree with their conclusions. And WP:WEIGHT requires us to respect the (scientific) consensus, which requires secondary sources to determine.

It is clear that Cochrane papers can be a good source of evidence. But where the review eliminates so much and find that what is left is insufficient to make a positive statement, we must treat it with caution. It could simply be that this "we know nothing => can't recommend" is an artefact of their rather extreme methodology, and actual clinical practice says otherwise. Where their papers turn out to be written by non-experts or deviate from performing a systematic review of the published literature, we must treat it with great caution. But ultimately, their papers are written to influence medical opinion and be a good source of evidence for experts to make decisions. But it is that medical opinion and expert decision-making that we report. -- Colin°Talk 11:46, 7 December 2014 (UTC)

Proposed changes

1. Proposed change

-A1candidate (talk) 10:06, 30 November 2014 (UTC)

2. WP:List of reliable medical sources (examples)

-A1candidate (talk) 10:06, 30 November 2014 (UTC)

3. Discussion

Proposed changes are probably more concise and informative. -A1candidate (talk) 10:06, 30 November 2014 (UTC)

There is quite a lot in medicine that can never be ethically or practically subject to randomized controlled trials. There are many more topics that could theoretically be but are currently not the subject of any sham- or active-controlled RCTs, many common surgical procedures included. I'm also not sure why it would help to have this short list of reliable medical sources. We have WP:MEDRS#Choosing sources; if people aren't already familiar with the biomedical literature and also aren't willing to spend the time to read that section, then they probably ought not to be editing in the first place. What would the effect of these proposed changes be on the guideline (from a practical or any other perspective). NW (Talk) 16:06, 30 November 2014 (UTC)
The list is just an example and is meant to be expanded upon if there's consensus to include something like that. WP:MEDRS#Choosing sources includes very few examples and isn't exactly helpful. Many specialty journals and important medical societies are not even mentioned. -A1candidate (talk) 17:05, 30 November 2014 (UTC)
Yes of course not. We are never going to create a comprehensive list of reliable sources. But it also becomes pretty obvious as soon as you start looking into a research field what's obvious and what isn't. Someone brought up a dispute on pain medications at Talk:Opioid on WT:MED recently. I've been looking into that independently on my own time recently. Just for non-cancer chronic pain, no fewer than ten professional societies have published guidelines. These include the International Association for the Study of Pain, the European League Against Rheumatism, the American Geriatrics Society among many others. I hate buzzwords, but medicine truly is more interdisciplinary than ever before—if you try to enumerate what is a good source and what isn't, you won't succeed. I'm still not really seeing what is broken about the current guideline, and I fear that your proposals will cause more problems than they will solve. NW (Talk) 21:02, 30 November 2014 (UTC)
Agree with NW, it's not a good idea to try to specify what exact kinds of evidence authoritative medical organizations must use before we can use what they say. I don't agree either with going down a path that looks like (even if it's not meant to be) "The Official List of WP:MEDRS-Compliant Sources", with the likely unintended consequences of "My source is on this list so everything they published must be included. Your source isn't on this list so it may never be used." ... Again, well-intentioned ideas but some serious negative unintended consequences likely. Zad68 19:07, 30 November 2014 (UTC)
Zad, the purpose of this list is simply to move all the examples that are already listed in the following statement: "... in statements and practice guidelines issued by major professional medical or scientific societies (for example, the European Society of Cardiology or the Infectious Disease Society of America) and governmental and quasi-governmental health authorities (for example, AHRQ, USPSTF, NICE, and WHO)..." and put them in a seperate place so that things don't look so cluttered. Either we create a new section for these examples, or we remove them altogether. I don't think it's a good idea to insert these examples in the middle of the sentence. -A1candidate (talk) 20:25, 30 November 2014 (UTC)
But inserting it into the text is done for precisely so that people can read the examples in proper context. I want people to be citing the CDC, NICE and the IOM. But I also want editors to actually read the whole guideline so that they know that "The reliability of these sources range from formal scientific reports, which can be the equal of the best reviews published in medical journals, through public guides and service announcements, which have the advantage of being freely readable, but are generally less authoritative than the underlying medical literature" rather than simply being able to parrot "IOM=best possible source". If we want to better integrate the "summarize scientific consensus" section with "choosing sources", I have no objection, but I don't think your proposal is the best way to go about it. NW (Talk) 20:50, 30 November 2014 (UTC)
I read A1candidate's proposal, and the words rare disease kept echoing in my mind. This is a fine ideal when you're talking about the treatment of hypertension or something similarly common, but it simply does not work for rare diseases. There are pretty much zero randomized controlled trials for very rare conditions, because there aren't enough patients available to run the studies.
Also, it's overly focused on treatment efficacy, when treatment ought to be maybe 10% or so of a well-developed article. It just doesn't work for anything except treatment and diagnostic tests. Has anyone here ever seen a meta-analysis of the definition of autism? How about a randomized controlled trial to figure out which definition is better? A systematic review on the prognosis? Me neither, but we could write an entire article about that subject—just, you know, not by using statements "backed up by randomized controlled trials and systematic reviews with meta-analyses". This isn't going to be functional for 90% of what we need to write. WhatamIdoing (talk) 13:01, 1 December 2014 (UTC)
On the disappointing evidence base for parachutes as an intervention

MastCell pointed to a well-known humorous essay Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. There is of course a grain of truth in the humor, but to see how we might handle it: Every medical society around the world strongly recommends parachute use for those who jump out of airplanes(cite review of practice guidelines worldwide), although no trials of the type that can produce the best kinds of evidence have been conducted into parachute use, due to ethical concerns in conducting those kinds of trials.(cite Cochrane) Yes, taking reductio ad absurdum for face value is even more absurdum but as before there's a grain of truth in it. Zad68 19:34, 30 November 2014 (UTC)

There is wisdom in the saying "absence of evidence is not evidence of absence".
There are areas in clinical medicine where trials might not be ethical (e.g. surgery for cancer or anticoagulation for large pulmonary emboli), even though the treatment was widely introduced before RCTs could be conducted. The parachute in the example is of course a very extreme example of the same phenomenon.
On the whole, if that is the scenario we're dealing with, it should also be possible to find a secondary source to explain why RCT evidence is unlikely to ever appear. As such, I don't think we should be including it in our arguments. JFW | T@lk 19:46, 30 November 2014 (UTC)
JFW yes, we're in agreement here. Did my comment not express that? Sorry for being unclear if not. Zad68 19:55, 30 November 2014 (UTC)
I didn't intend to dispute your post. It is a rephrase, and I think we both agree that it is possible to mention "established practice" with the EBM view side-by-side. This is not an uncommon scenario - NICE guidelines have a habit of discouraging practices that everyone believes to be normative. JFW | T@lk 19:59, 30 November 2014 (UTC)
I think there is an element of truth to this.
  • Repeatedly cited in Cochrane reviews of antipsychotics is the statement that "there is little evidence for efficacy because of high drop out rates in clinical trials". I would say that trials meeting Cochrane's requirements are unlikely to appear because the dropout rates are mainly a characteristic of the disease and not the medication. How will one design a drug or trial that reduces dropout in the placebo arm? Cochrane might as well stop doing meta analyses on antipsychotics now, because the result will always be the same.
  • In the paliperidone ER trial NCT00396565, the completion rate was 65% in the paliperidone arm, 63% in the olanazapine active control arm, and 34% in the placebo arm
  • In the Aripiprazole LAI trial the completion rate across the 3 doses of active drug were 75% vs. 51% in the placebo arm
  • In the Ziprasidone trial in adolescents NCT00257192 the completion rate was 70% in the active arm and 57% in the placebo arm.
I've looked for a long time for a secondary reference on the difficulty of conducting a trial that will meet Cochrane's standards, or even one that points out that high discontinuation rates are found in the placebo arm more so than the active one, but without success.
  • No one running an antibiotic trial will allow a patient to die just to collect a survival endpoint. Those failing treatment will be switched to a different antibiotic. I don't know where I would look for a reference stating this. Nonetheless, there are meta analyses out there that have concluded that fluoroquinolones don't reduce mortality in hospitalized CAP patients, and that they don't reduce mortality in HAP. I'm not sure how one would demonstrate this when those in both arms with non-responsive infections are rapidly moved to other Abx.
Where are the RCTs demonstrating that being obese and eating red meat are bad for your heart, that exposing oneself to cigarette smoke or even asbestos will give you cancer, or that drinking unchlorinated water from a river downstream from an untreated sewage outfall puts you at risk of GI infections, etc? These are all pretty widely accepted ideas, showing that we accept non-RCT data to support many of the most important of our public health priorities. No one has conducted a placebo controlled trial in bacterial pneumonia since the dawn of the antibiotic era. Formerly 98 (talk) 20:46, 30 November 2014 (UTC)
Alot of psychiatry is tricky - any prospective study on suicide is difficult due to the rarity of the event, and psychoanalysis just takes so damn long and is also uncommon. Much better concerted work could be done on antipsychotics - I have read alot of trials that were unsatisfactory for entirely avoidable reasons (e.g.comparing new drug to very high doses of haloperidol and then noting the lower rate of EPSE of the new drug. Studies comparing new drugs to placebo rather than to each other etc. "pre-selection" with quetiapine (see talk page archives) yada yada) Cas Liber (talk · contribs) 03:49, 1 December 2014 (UTC)
Oh, I don't think anyone is going to argue that drug companies design clinical trials to provide the maximum decision-making power to healthcare providers. I certainly won't anyway. They'll do that when they are sure they have a superior drug on their hands and not otherwise. And psychiatric drug development and marketing bears the same relationship to drug development generally that science fiction does to the field of literature. Its mostly an embarrassment.
But I do think we know a lot of things that don't come from RCTs. And given the high and probably unfixable dropout rates in the placebo arms of antipsychotic trials, I can't begin to imagine why Cochrane continues to do meta analyses in this area if they see that as a problem. They'll be drawing the same "dropout rate is too high to make any interpretation of the data" conclusions 100 years from now that they make today. You can't fix dropout in the placebo arm with a better drug. Formerly 98 (talk) 04:10, 1 December 2014 (UTC)
You can fix it with a better health care system like the Scadinavian ones. Doc James (talk · contribs · email) 05:09, 1 December 2014 (UTC)
I'm not sure that would help.
  • Sadly part of what drives participation in trials is lack of access to good healthcare outside of trials and or financial incentives. It is pretty hard to recruit schizoprhenic patients to a trial in which they may recieve placebo in the context of a good social welfare system where they have some sort of guardianship, financial security, and good healthcare. I have a vague memory of looking into this once and finding a bunch of Swedish trials that were terminated for slow enrollment, but don't remember the details.
  • The most important driver of discontinuation in these trials is lack of efficacy, as indicated both by investigator reports and the lower dropout rates with more efficacious treatments. So you really can't keep patients on placebo except by unethical coercion
  • Once they drop out, randomization is lost and so is blinding, no matter what records are kept. Formerly 98 (talk) 09:39, 1 December 2014 (UTC)
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:Identifying_reliable_sources_(medicine)/Archive_11&oldid=1137415731"