User:Shibbolethink/Sandbox
RFC: ProPublica
[edit]From a 2019 RFC, ProPublica at WP:RSP is summarized as: strong consensus [of]... generally reliable for all purposes because it has an excellent reputation for fact-checking and accuracy, is widely cited by reliable sources, and has received multiple Pulitzer Prizes.
This has been tempered by several recent controversies.
1) COMPAS algorithm debate:
PP did a report in 2016 on the COMPAS algorithm, made by a company called Equivant, used to "automate decision making for bank loans, school admissions, hiring and... predicting recidivism" Importantly, race is not used as an actual predictive factor in the algorithm, though as most statisticians know, it can often be an unintentional confounder. [1] As Technology Review notes: When ProPublica compared COMPAS’s risk assessments for more than 10,000 people arrested in one Florida county with how often those people actually went on to reoffend, it discovered that the algorithm “correctly predicted recidivism for black and white defendants at roughly the same rate.” But when the algorithm was wrong, it was wrong in different ways for blacks and whites. Specifically, “blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend."
After publication, a number of controversies arose about the reporting. As a report in Washington Post authored by several statistics and data science professors notes, "Northpointe, the...company that created the tool, released
its own report questioning [the] analysis. ProPublica
rebutted the rebuttal, academic researchers
entered the fray, this newspaper’s Wonkblog
weighed in, and even the Wisconsin Supreme Court
cited the controversy in its recent ruling that upheld the use of COMPAS in sentencing.
WaPo goes on to say, citing statistical principals, that it’s actually impossible for a risk score to satisfy both fairness criteria at the same time.
Because it's an unfortunate reality that [t]he overall recidivism rate for black defendants is higher than for white defendants (52 percent vs. 39 percent).
[2]
In particular, criticism was levied at PP's analysis. The Crime and Justice Institute's experts published a report highlight[ing] the flaws and erroneous assumptions made in the ProPublica article and us[ing] the data employed by the Larson et al., to conduct an independent analysis. Findings revealed that the conclusions reached by Larson et al were not supported by the data. Using well-established research methods, the authors failed to find any evidence of predictive bias by race in the COMPAS data used in the ProPublica article.
In particular, the CJI authors (and independent experts) criticized the PP authors' understanding of...risk assessment
, how the algorithm is actually used, and PP's understanding of research methods and statistics
and due diligence
. The CJI isn't some super biased political think tank, either. They're a 136-year-old nonpartisan group based in Boston, trusted by multiple municipalities and govt agencies across the spectrum to produce unbiased reports on how justice is served, including: Milwaukee, WI, Dept of Justice [3], Yolo County, Maryland, the states of South Dakota and Nevada, and Pew Charitable trusts. This is exactly the sort of organization we would want to be judging PP's analysis. And they deemed it unacceptable in its conclusions.
2) COVID origins investigation: In November of this year, ProPublica published a jointly reported piece with Vanity Fair about a senate Republican minority staff interim report on COVID origins, which was leaked/released right before the midterm elections. The piece claimed that certain
Almost immediately, the report was met with numerous criticisms. For one, (linguistics issues). For another, (issues with november vs august dates as described by FP deputy editor). And most of all, (misreporting or downplaying of criticisms).
Michael Hiltzik, writing for the L.A. Times, called the reporting a train wreck
. [4] The Guardian.... Semafor... NYT... Science...
So what are we to do? Should we change ProPublica's listing on WP:RSP?
- Option 1: Generally reliable for factual reporting (status quo)
- Option 2: Unclear or additional considerations apply
- Option 2.1: Should be attributed
- Option 2.2: May be unreliable for science and data-based investigations
- Option 3: Generally unreliable for factual reporting
- Option 4: Publishes false or fabricated information, and should be deprecated