User:Novem Linguae/Essays/Studies are not reliable

A "study" sounds very reliable, trustworthy, and authoritative. They're done by scientists and go through peer review. So that's reliable, right?

It's counter-intuitive, but single studies really aren't reliable. In fact, a lot of them are junk. There is something in science called the replication crisis. Someone will try to reproduce a study with the exact same conditions as the original, and will often get a completely different result. This is a huge red flag that the methodology of the original study was poor.

Being reproducible is a core tenet of empiricism and the scientific method. If a study is not reproducible, it likely suffered from poor methodology and its results are junk.

In the vast majority of cases, a single study is a starting point, not proof of anything. The results could be random chance, or the result of bias, or even outright fraud. Only once other researchers replicate the results can we consider a study persuasive. The more replications, the more reliable the results are. If attempts at replication fail, this can be a sign the original research was biased or incorrect.

Methodology

Examples of poor methodology and other red flags that can affect the accuracy of a study's conclusions:

• Sample size too small - The results are explainable by random chance
• Not double blind - The researcher's confirmation biases may affect the collection of the data
• Lack of a control group / placebos
• Has a conflict of interest - Is the maker of cancer drug cancer-be-gone funding a study that happens to conclude that cancer-be-gone works wonders? Is the fossil fuel industry funding a study that concludes that global warming isn't real?
• Isn't peer reviewed
• Isn't published in a reputable journal

What about newspaper articles that report on a study?

Newspapers often just regurgitate studies that have sensational, counter-intuitive conclusions. This is part of sensationalism and click bait. Reporting on a study with a counter-intuitive conclusion will get clicks for the newspaper article, even if that study's conclusions are completely false.

Examples of click bait article titles based on junk studies:

• "Taking aspirin three times a week boosts chances of beating breast and bladder cancer by up to a third, study finds"
• "Cheap antidepressant shows promise treating early COVID-19"
• "Natural immunity IS more powerful than vaccines, another study hints: Antibodies in un-jabbed Covid survivors are 'stronger' over time than in people who've had two shots but no infection, study claims"
• "Ebola can 'hide' in a person's brain after infection and kill them YEARS later, US Army study finds"

Junk newspaper articles based on junk studies can occur in normally very reliable newspapers. The second bullet above was an article by the Associated Press. Because even normally reliable newspapers run articles based on junk studies, newspapers should not be used as citations for academic information at all.

Review articles

WP:MEDRS, a set of stricter sourcing requirements for biomedical information which outright bans single studies as references, was probably created at least in part to keep these inaccurate studies and inaccurate newspaper articles out of Wikipedia.

Over time, Wikipedia's science editors have figured out that the most accurate scientific knowledge is found in 1) review articles, 2) textbooks, and 3) statements by national and international science organizations. Therefore, this is what MEDRS says and this is what we use.

The above three types of sources all have in common this: an additional layer of experts who have read multiple studies on the same topic, and used their expertise to evaluate which studies have accurate conclusions and which studies have junk conclusions. This additional layer of screening makes these sources much more reliable. These sources are all WP:SECONDARY or WP:TERTIARY, unlike studies which are WP:PRIMARY.

Scope

MEDRS is a Wikipedia guideline and is required for medical sourcing. However the idea behind it (that the best scientific sources are review articles, textbooks, and statements by national and international scientific organizations, and that single studies are unreliable) applies to all studies and all scientific sourcing. In fact, there are spinoffs of MEDRS for other fields, for example, WP:SCIRS for science.

The bottom line is this: Wikipedia editors should use extreme caution when adding references to single studies in any topic, and should try to avoid doing so at all. In addition to the above reasons, single studies are always WP:PRIMARY, and are often WP:UNDUE. WP:REDFLAG applies to studies with counter-intuitive, sensational, extraordinary conclusions.

See also

• User:Novem Linguae/Essays/MEDRS simple explanation