User:Tom Morris/The Reliability Delusion

Let us meditate upon Wikipedia:Identifying reliable sources, a content guideline on Wikipedia that we all consider pretty fundamentally important to Wikipedia's mission. Finding reliable sources is a more fundamental research skill than just Wikipedia though: in academia, in journalism, on the sister projects (Wikinews, Wikiquote, Wiktionary and Wikispecies). If you want to say something is a fact, one good way of doing so beyond direct empirical observation, the scientific method (direct observation's older, wiser, more self-critical sibling), or logical deduction from a priori principles (mathematics, logic, various branches of philosophy and much else) is to appeal to the testimony of others. And not just anybody, but testimony of people who can be relied on to give you something closely approximating reality to a close enough degree.

But, as philosophers throughout the ages have pointed out, the idea that testimony is a source of knowledge often has problems. Not insuperable ones, mind, but problems nonetheless. These problems all tend to be caused by an unfortunate fact about human beings: they lie, they misrepresent, they bullshit and they often just misremember and fail in other unfortunate ways. It is for this reason that we have to critically evaluate what people say. This applies to representations of what they say too: and it is from this that we need to try and evaluate sources of information for quality.

The naïve view

Imagine you have two buckets before you. Every possible source you might use in a Wikipedia article (or an essay for school or university, or a news report, or a blog post or whatever) must be placed into one of two buckets. Bucket one has the word reliable printed on the side, and bucket two has the words not reliable printed on it. Choose wisely.

An objection

In the above scenario, we end up believing that there is a binary or at best simple, unary property of reliability. This analysis gives us sentences of the form:

x is a reliable source.

y is not a reliable source.

There is something fundamentally wrong here.

Meet John. John is a human being and as such he likes to have beliefs about the world and occasionally he likes to make statements that reflect his beliefs. He is deeply committed to telling the truth and never knowingly lies with one very important exception: whenever he visits Chicago, he is unable to correctly identify the name of politicians. If you ask him who the President of the United States is when he is anywhere other than Chicago, he'll give you a good response, but you take him to Chicago and he'll tell you that Maurice Gibb from the Bee Gees is the president rather than Barack Obama. Very occasionally, just by accident, or by some kind of Gettier fluke, he'll get the answer right, but not because he is giving you an accurate testimony to a successfully attained instance of knowledge under, well, pretty much any epistemological theory you choose to mention: it hasn't been reliably produced (per David Malet Armstrong, Alvin Goldman etc.), it hasn't been the result of a causal process, it doesn't "track truth" such that if p, S believes that p and if not p, S believes that not p (per Robert Nozick), the person isn't aware of his knowledge basis for his belief or is unable to come to see his knowledge basis by a process of reflection (the various internalists), he has fallen short of his epistemic duties or failed to be an epistemically virtuous person etc. etc.

You have to forgive poor John. It's just a strange little quirk: poor John, let's hope his health insurance covers the promising neural implant various philosophers hint might be available to fix his highly context-specific epistemic errings.

You just made that up!

Okay, let's take a source many people consider reliable: The Guardian and The Observer. As a British left-wing type, I'm a big fan of the Grauniad and the Observer, and as newspapers go, I consider them pretty reliable. But if we are using the naïve view of reliable sources, I would be unprepared to put them into either the reliable or unreliable bin.

Why?

The Barefoot Doctor. For a long time, the Observer had a health column in their glossy supplement from Stephen Russell aka. The Barefoot Doctor. In said column, he would answer readers advice questions with a truly bizarre range of crackpot, looney and wacko pseudoscientific solutions: stand on your head, chant the word "ummmmmmmmm" meditatively, use a homeopathic suppository while pouring vegan soy gravy in your left ear and all your woes will be cured. Even by the incredibly low standard of alternative medical advice, it's pretty damn strange.

A famous example of a bizarre Barefoot Doctor remedy suggested to a woman asking about tinnitus:

Tinnitus, unless caused by physical damage to the eardrum by loud noise, originates from low kidney energy, as the ears are said to be the flower of the kidneys.

The trauma of your son's death must have significantly weakened the kidney energy.

Press your thumbs into the ridges of muscle that run down either side of the spine around the kidney region (in the lower back) every day for a minute or so, especially in the late afternoon, when the kidneys are most susceptible. Also massage the ears themselves quite vigorously between thumb and finger to increase circulation. Consider visiting an acupuncturist, as tinnitus responds well to acupuncture.

Is this a reliable source? Again, pick your bucket: if you say The Observer is a reliable source, do you want the above material being included in the article on tinnitus? If you say The Observer is an unreliable source because you think "weakened kidney energy" is a crackpot theory, that's fine, but the publication of our Barefooted friend doesn't discount the good work done by reporters covering real stories. If the only question asked of a source is simply "Is this a reliable source, yes or no?" then we actually aren't doing nearly enough in working out whether the source is appropriate.

Works the other way too

When we say that a source is unreliable, it is another way of saying that we consider that source unusable for anything. Let's take for the sake of argument that there are sources we consider unreliable for telling us the truth about most aspects of the world. And let's also take for the sake of argument that creationist sources count for that. Scientists rightly believe that the world is not six to ten thousand years old. The jury really is in on this. What possible use could there ever be for the website of Answers in Genesis as a source for anything given this fundamental reality-denying premise upon which everything they do is based?

Simple.

The Answers in Genesis website is a reliable source for telling us what the people at Answers in Genesis believe. Furthermore, it is a reliable source for telling us that the Creation Museum they run is indeed located in Petersburg, Kentucky, and it tells us that Ken Ham is President of the group. There are countless other things that their website is reliable to tell us, and we should accept those things as reliable unless we have good reason to see a conflict, in exactly the same way that if you ask me while I am walking along the street in London to tell you where Buckingham Palace is, you should probably trust me unless you have reasons to doubt what I say—if I tell you to get on the train to Heathrow Airport and fly to Las Vegas, that is a pretty good reason to doubt what I say and go find someone who isn't crazy and ask them for directions!

Back to the guidelines

Wikipedia:Identifying reliable sources has an oft-overlooked statement worth reflecting upon:

Proper sourcing always depends on context; common sense and editorial judgment are an indispensable part of the process.

It is my firm belief that if we had a more mature way of talking about sources, we would have fewer problems.

Context sensitivity

Individuals vary in what kind of knowledge they have access to. They have different background knowledge, different experiences, different capabilities, different biases. This is reflected in the sources they produce.

When we communicate about sources at the moment, we use sentences that are of the form:

x is a reliable source.

But if we started introducing some other variables into those sentences, we could do much better:

x is a reliable source on topic t.

x is a reliable source for material of genre g on topic t but not for topic t₁, t₂ and t₃.

x is a reliable source for most things but not t₄ where it is frequently unreliable.

x is not a reliable source for most claims but they are good enough for claim c.

When someone follows the naïve formulation of reliability, tell them: more context, more editorial judgment and more common sense can be achieved very easily by being more specific about the contexts and limitations of the reliability of a source.

To give the example from above:

The Observer is in general a reliable source for news and current affairs, but the presence of the Barefoot Doctor column means that we can't trust the health advice in the glossy supplement magazine that comes with the paper.

This is far more helpful than simply trying to determine if The Observer is a reliable source or not as it tells you exactly the limitations and reasoning that goes into making a decision about any particular use of The Observer in an article.

Indeed, this is the message Wikipedians frequently try and point out to people who use Wikipedia and people who refuse to use Wikipedia because of concerns about it being "unreliable". Well, reliability for what? For the sort of disputes you might use a general encyclopedia to determine the answer for, Wikipedia is pretty damn good—maybe not perfect, but perfection is a fool's errand anyway. If you wake up in the middle of the night and you want to what the atomic number for Rhenium is, Wikipedia's failure to live up to Truth With A Capital T-levels of perfection—or indeed, some arbitrary and ad hoc standard established for what is reliable–shouldn't bother you, and in exactly the same way, a source doesn't have to be perfectly reliable for it to be pretty damn useful. This requires little more than applying common sense.

"Pretty damn useful" may be a cheap substitute for perfection, but as Daniel Dennett has said of conceptions of free will that aren't contra-causal, "try the cheap substitute, it's pretty good!" For all but the easiest questions, the quest to find reliable sources will lead into murky hinterlands where one must simply cast one's bets and hope for the best. If you expect absolute perfection from sources, you'll end up having very few sources.

Practical applicability

Could we apply these common sense reflections to current policy? Probably not directly. At best, we can use them to find some inconsistencies and problems with our current rules over sourcing and try to fix those. As we go forward, we can stumble into bad policy less often if we understand that the reliability of sources isn't a simple binary matter.

Blogs aren't evil

Wikipedia has a rule against using blogs as well as other social media sites as sources. This strikes me as irrational. Blogs are just a type of content management system, like wikis or Drupal or Joomla or Slash (the engine used by Slashdot). Some blogs are run by very reputable sources and can be pretty trustworthy. Are we saying that ceteris paribus, OUPblog loses points because it is a "blog" but TimeCube doesn't because at least the TimeCube's brand of crazy isn't delivered using WordPress or Moveable Type? Even if TimeCube were to use blogging software, it's still crazy, and the material on OUPblog should still be considered pretty reliable even if they decide to change what type of software they use to publish that material on the web.

POV sources aren't evil

We try hard to avoid using "POV" sources: in general, we would rather use sources that present a neutral reporting of the facts rather than an opinion piece. If it's a choice between Alternet, RedState and CNN, we'll use CNN, right? But what is wrong with using POV sources to establish facts?

Well, there are two possible issues: one, it may give the appearance of impropriety. If you go to the article on, say, Barack Obama and it's all citations from sources pushing the same POV, it may feel like a put-up job by POV-pushing supporters or opponents using the article to push their politics. Secondly, the POV sources may get the facts wrong.

Allow instead the use of POV sources to establish facts in minor cases. I created the article Maurice Généreux, about a Canadian doctor who prescribed lethal drugs for his patients. I'm using a pro-life source to establish his prison sentence. Is this a big deal? Not at all. I'm not using the pro-life source to say that Généreux was an evil, kitten-eating monster, just to establish a fairly neutral fact. I'm not even mentioning that the pro-life community who run the site in question have a problem with what they believe to be a short prison sentence. It's an uncontroversial fact, it shouldn't have the appearance of a "put-up job" or POV pushing, and the facts in this source line up with reporting elsewhere.

If you are only able to think "reliable or unreliable", a self-published source from an advocacy group would fall into the "unreliable" category even though the job it is serving in this article is just fine.

And, no, I didn't add the source to Maurice Généreux to prove a point, it just happened that afterwards I realised it proved my point.

Where now?

How should the community work on policy related to sourcing and the reliability thereof? We must keep in the front of our mind that coming up with a method to sort reliable sources from unreliable sources will never be simple or straightforward. Moses will not come down from the mountain with divine guidance on sourcing issues. All decisions about the reliability of sources will eventually end up being context-specific judgment calls. Trying to come up with a way of making those judgment calls in a neutral, consensus-driven manner may be more difficult than the content guidelines suggest.

It is customary to end essays of this sort with some practical, uplifting advice. I should probably adhere to that custom.

When faced with a question of whether a source is or is not reliable, evaluate the context and ask if the source would be considered by a reasonable person trained in the field of life to be an appropriate, useful and interesting contribution? Try to balance WP:RS with common sense, and with other content policies like WP:UNDUE and WP:FRINGE. If in doubt, try and get a third opinion.