Wikipedia:Reliable sources checklist

The goal

• What are we trying to do here?
  • A succinct description of what we want to find out, e.g. "We want to find out what the actual area of Baltimore is" or whatever.

The material

• What's the material that the source supports?
• Is it contentious or contested?
  • Contentious material is material that people might take a position on for ideological reasons. Contested material is just material that another editor has made a reasonable challenge to or where sources disagree. (Whether a person was born on April 19 or April 20 might be contested but it's not an ideological issue.) If it's not contentious, the source is only going to be wrong because of failure of diligence. If it's contentious, we also have to be aware of the possibility of deliberate bias.
• Does the source indeed support the material?
  • For instance, if it supports a quote, do the quoted words indeed appear in the source, and so forth.

The author

• Who is the author?
• Does the author have a Wikipedia article?
  • This provides a quick-and-dirty (albeit imperfect) gauge of notability, which is not the same as veracity but is a data point. And the article and its links provide a good start in figuring out who the person is.
• What are the author's academic credentials and professional experience?
• What else has the author published?
• Is the author, or this work, cited in other reliable sources? In academic works?
  • This is a rough indicator of post-publication peer review and acceptance. (Pro tips: if there is a DOI link, this will often show "where cited" for scientific publications. If it's a book, searching at Google Books (enclose the book title in quotes) can bring up cites.) WP:NJournals describes some other citation-finding tools and methods.)
• How does the author make a living?
  • If they work for a salary, they have an incentive to not get fired. That means if they are a professor at an established university and that's their main source of income, they have an incentive to avoid outright mendacity, since that'll get you fired. Newspaper reporter, same thing. But other entities might encourage mendacity if it supports their mission. If they make a living writing books or whatnot, their main financial incentive is to increase sales rather than necessarily get their facts right.
• What about reputation? Are there any big character markers?
  • If they have been fired for plagiarism or indicted for perjury or successfully sued for libel or whatever, those are data points. It's probably best to be skeptical about other markers. Matters unrelated to their writing such as sex scandals might or might not indicate anything. Awards and accolades might matter some, depending on the source, but it's probably best to avoid giving much importance to man-of-the-year type logrolling. Most everyone has enemies, so a lot of people have been called a liar by somebody. If there's a pattern of disinterested people doing that then it might indicate a problem.
• Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
  • Even if they are utterly disinterested, they can still be inaccurate, of course. But if they are well to the right end of this continuum, that's a big red flag. It doesn't mean everything they say is inaccurate, of course, but it's an important data point. You have to be honest here – if they are a polemicist who supports your version of things, they are still a polemicist.
• Anything else?

The publication

• What is it?
• Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
  • WP:RS, in its sections WP:SCHOLARSHIP and WP:NEWSORG, strongly (and sensibly) indicates that these are the only sources that are assumed to be reliable. (This doesn't prove that they are reliable in a given case, just that the assumption that they are is your starting point.) Everything else is up for debate. WP:RS is lengthy and there's a lot of hedging, but that's a reasonable summary.
• If not, is there any reason to believe that anyone has checked the author's facts?
• What's their circulation?
  • Size doesn't prove anything, but it's a data point. The New England Journal of Medicine and the North Carolina Literary Review are both scholarly journals, but they're not equal. Ditto the New York Times and the Easton (Maryland) Gazette. A bigger operation means more resources for fact-checking, a bigger reputation to uphold, and greater likelihood of employing top-tier people.
• What about the publisher? What kind of outfit are they? What's their reputation?
• Do they have an agenda?
• What's their business incentive for veracity?
  • Some magazines and newspapers rely on their reputation for veracity as part of their marketing model. If they don't pay attention to that they're eventually out of business. Others, not so much.
• Anything else?

Other

• Does the source have standing to address the material?
  • Does the source have the expertise on the particular topic being ref'd?

The biography Cary Grant is by the popular, prolific, and well-reviewed expert Marc Eliot. Most editors would assume that his book is a reliable source. But... from the book: "Screwball is a particularly apt term for a certain type of movie that, like the baseball pitch of the same name, travels a fast but unpredictable path before somehow managing to cross the plate for a perfect strike." But that's not an accurate description of a screwball pitch at all; not even close. Eliot has standing to be used as a source on Cary Grant, but he doesn't have standing to be used as a source on American baseball.

• Anything else?

Blank checklist (to copy and use)

Example #1

The goal

• What are we trying to do here?
  • Determine if Gregor Mendel's work was the foundation of genetics.

The material

• What's the material that the source supports?
  • Mendel demonstrated that the inheritance of certain traits in pea plants follows particular patterns, now referred to as the laws of Mendelian inheritance. Although the significance of Mendel's work was not recognized until the turn of the 20th century, the independent rediscovery of these laws formed the foundation of the modern science of genetics. As with many cites, it's sometimes not clear exactly what's being ref'd – it might be only the second sentence.

• Is it contentious or contested?
  • No, neither. We're just using this an example. (If the material described Mendel himself as "the founder of genetics" or something, this could be contested, since Mendel's laws were forgotten and probably not much would be different if he'd never lived. However, the material is careful not to make this claim.)

• Does the source indeed support the material?
  • It seems to. It's a book, and it's not online, and no page number is given, so this makes it hard to judge using the limited preview that Google Books provides. The ref'd material discusses Mendel at length and the material in the article is a severe reduction and simplification of this. It looks to be a reasonable extrapolation from the ref'd material.

The author

• Who is the author?
  • Peter J. Bowler. He's a historian of biological science and a professor at Queen's University Belfast.

• Does the author have a Wikipedia article?
  • Yes, he does, Peter J. Bowler.

• What are the author's academic credentials and professional experience?
  • He has a Ph. D. from the University of Toronto. He's worked as a teacher or professor. He's a past President of the British Society for the History of Science.

• What else has the author published?
  • A number of scholarly works. His article lists some of them. He's written a textbook and a few other books. Judging just by the titles these look to be high-end general-interest books for the educated layman.

• Is the author, or this work, cited in other reliable sources? In academic works?
  • Yes, some. It's listed as a ref in J. L. Heilbron's The Oxford companion to the history of modern science and some other works.

• How does the author make a living?
  • He's a professor at Queens College. This is probably his primary source of income, although he also gets book royalties.

• What about reputation? Are there any big character markers?
  • Don't know of any. He's a member of good standing of various scholarly bodies, according to his article.

• Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
  • He's an academic. There's no reason to believe that he'd have any incentive to either glorify or deprecate Mendel's role. He is a strong public voice against creationism, but 1) that shouldn't affect his regard of the standing of Mendel in particular as opposed to other figures, and 2) that's not really being polemic, that's just being a biologist, as essentially 100% of reputable biologists are opposed to creationism.

• Anything else?
  • No.

The publication

• What is it?
  • It's a book: Bowler, Peter J. (2003). Evolution: the history of an idea. Berkeley: University of California Press. ISBN 0-520-23693-9.

• Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
  • No.

• If not, is there any reason to believe that anyone has checked the author's facts?
  • Not sure. It's University of California Press and that's a reputable outfit. Whether they check Bowler's work or just trust him I don't know. Probably the latter.

• What's their circulation?
  • Not applicable. UC Berkely is a big school and their publishing arm is also a big operation.

• What about the publisher? What kind of outfit are they? What's their reputation?
  • University of California Press, Berkely. It's a highly reputable academic publisher.

• Do they have an agenda?
  • No.

• What's their business incentive for veracity?
  • It's high, as far as that goes, since they're an academic publisher.

• Anything else?
  • No.

Other

• Does the source have standing to address the material?
  • Yes, certainly. He has considerable expertise in the subject.

• Anything else?
  • No.

Summary
It's an acceptable ref. It's a book, and we don't like to use books as sources, since books aren't usually fact-checked, so we are basically depending on Bowler's reputation. It appears to be excellent. There are no markers to indicate that he would have any incentive to get this wrong (quite the contrary, he has an academic reputation to uphold) and every indication that he has the competence to get it right.

About fact-checking

Most large magazines employ fact-checkers. Book publishers and most newspapers don't. They employ copy editors whose main brief is fixing grammar and style. Copy editors may check facts, but only on an ad hoc basis.

When you cite a book, you are relying almost entirely on the author. Book publishers have little incentive to worry about facts since people generally buy books based on the author rather than the publisher. For this reason books are seldom very reliable sources.

When you cite a newspaper, you are also relying on the author but mainly on the publication. Rather than checking facts, newspaper editors will expect reporters to check their own facts and they'll fire them if they don't and reporters know this. Newspapers do have an incentive to worry about facts since people do generally buy newspapers based (partly) on the paper's general reputation for veracity and not on the names of particular reporters. It depends a great deal on the newspaper, of course, and business incentives to get facts right varies a lot among newspapers, and so does editorial rigor.

Journalistic entities known to have good fact-checking operations

This list is incomplete; you can help by adding missing items.

But don't just throw your favorite paper into this list. A good citation describing their fact-checking operation would be helpful.

• Der Spiegel. According to the Columbia Journalism Review (2010) they have about eighty full-time fact-checkers – more than any other publication on the planet – and a very rigorous and well-organized operation.

But just to point out, any publication is vulnerable if a reporter is inclined to lie and falsify her notes. This has happened at the New Yorker and the New Republic also. From the crooked timber of mankind, no straight thing was ever made.

• The New Yorker is famous for the rigor of its editing (including fact checking). According to the Columbia Journalism Review (2010) they employ 16 fact-checkers. That's a lot for small weekly that doesn't specialize in hard news. Here's a New Yorker fact-checker recounting her service: "[A] long piece... received her full-time attention for three or four weeks... Each word in the piece that has even a shred of fact clinging to it is scrutinized... After an error gets into The New Yorker, heat-seeking missiles rise off the earth and home in on the author, the fact-checker, and the editor" Look at this: "Sometimes a phrase would contain hidden facts, as in 'Jane’s youngest son.' You’d have to check maternity and birth order, but you’d also have to confirm that Jane had at least three sons for one to be considered 'youngest.'" (If only two, it would be "younger", get it? These people are serious.) And "[T]he material originally appeared in The New Yorker, which, along with Time magazine, originated the practice of fact checking and has for many years been famous for the reliability of its content." - Ben Yagoda Here's from a 2012 piece in the Columbia Journalism Review: "To start checking a nonfiction piece, you begin by consulting the writer about how the piece was put together and using the writer’s sources as well as our own departmental sources. We then essentially take the piece apart and put it back together again. You make sure that the names and dates are right, but then if it is a John McPhee piece, you make sure that the USGS report that he read, he read correctly... Or if we describe the basis on which the FDA approved or disapproved the medical tests that ImClone used for Erbitux, then you need to find out what the complexities of that whole situation were."
• The Economist. (The Economist has a good reputation and a strong business incentive to get their facts straight since their core constituency includes high-powered decision-maker types who require reliable data and pay a lot for it, but so far we don't have a cite describing their operation, so... ^{[citation needed]}) There have been sporadic complaints about fact-checking at The Economist,^{[unreliable source?]} but as noted in The Atlantic, the Economist seems to employ fact checking in the book review process.
• British Broadcasting Corporation. The BBC has a rigorous set of guidelines including for accuracy, impartiality, fairness, and the reporting of controversial events such as war, terror and emergencies.
• PolitiFact deals with political statements. It has a business model of 1) checking facts – that's what they do and it's all they do, and 2) being strictly non-partisan. So based on this the assumption would be that they are fairly reliable. But we need more proof, so... ^{[citation needed]}

Journalistic entities known to have bad (or no) fact-checking operations

• Some Forbes material, specifically that found at "www.forbes.com/sites/". Despite the Forbes nameplate, these are blog posts and are not fact-checked at all. According to the Columbia Journalism Review, "Around 250 to 300 stories go up on the forbes site each day. ... No matter their background or compensation, all contributors can publish their own work without so much as a cursory edit.
• Newsweek is an old name in American journalism, but as their print operation died they fell upon hard times; the 2012 Niall Ferguson fiasco showed that, at least as of 2012, they weren't fact-checking at all. (Poynter,NYTimes) Since 2013 they are owned by IBT Media and we're not sure of if their reliability has recovered any or if so how much.
• Beall's List is a list kept by Jeffrey Beall of questionable and shady open-access scientific journals. Articles in journals on Beall's List are presumed not reviewed and should usually be treated as blog posts.
• Esquire, an established American magazine, does not come off too well in this anecdote (from Jacobs, A. J. (2005). The Know-It-All. Simon & Schuster. p. 48. ISBN 978-0743250627.): "I was helping edit an article... my boss had sent it back to me, suggesting it was a little bland... I simply tacked on [a] sentence [to a quote]... And it worked. My boss liked it better. The problem was, I completely forgot to send the piece back to the [person quoted] to see if she was okay with my little addition. I had meant to -- I know you can't just insert something without the writer's approval. But I forgot... [and later] the [subject] is complaining that Esquire put words in her mouth... I feel terrible... I'm hoping this little scandal will blow over, and I think it will."

On the one hand, the person knows they messed up and feels bad, and there was a "little scandal". On the other hand... why did this person even consider for one moment tacking on an extra sentence to a quote? And where was the fact-checker in all this -- where is the person taking the article from the editor and making sure the subject is called to verify the quote? There isn't one. And what is the meaning of "you can't just insert something without the writer's approval"? (If the article writer says "Sure, I don't care", it's OK then?) Where is the indication that person was put in fear for their career over this? I don't see that either. It's only one anecdote, but this is not a good look for Esquire.

Also, here is a 2017 Esquire article. It has spelling errors. So...

• People Magazine does have fact checkers, although possibly intern or junior new hire level I think, but then you have this anecdote which tells of a case of an article mentioning Abe Vigoda. A "top editor" added "the late" in front of Vigoda's name (Vigoda was alive). The fact checker took it out, but the editor insisted, and it was published in the article. (FWIW I think this was the start of the "Abe Vigoda is dead" meme which became rather widespread.)