Wikipedia talk:Wikipedia Signpost/2023-05-08/Recent research

Discuss this story

It's always interesting to read about Wikipedia policies. I myself was unfamiliar with TOOSOON, but it made me chuckle. Just to check, I ran a Wikidata query for the human genders male and female with sitelinks on English Wikipedia who were born after 01-01-2000. Of a total of 9049 humans, women represent 30%. Either it's been too soon for their articles to get Wikidata items that include their gender, or it's too soon for them to have an article, but women up to age 23 are still underrepresented on English Wikipedia by a large margin. Jane (talk) 05:43, 8 May 2023 (UTC)[reply]

From what I recall from research I did on this few years back, the more recent the time range considered, the less women are underprepresented. --_{Piotr Konieczny aka Prokonsul Piotrus| reply here} 08:27, 8 May 2023 (UTC)[reply]

Jane023, I am surprised that women represented 30% of your sample; I would have expected it to be lower, given the undue weight that Wikipedia authors and the results of our notability guidelines give to sports figures, who are predominantly male. I would be interested to see the same percentage breakdown for news coverage or other significant coverage in independent, reliable, secondary sources. In a world dominated by patriarchy, would you expect 51% of coverage to be about women? I would not; our societies have more progress to make in ensuring that the potential contributions of all people are valued and available. – Jonesey95 (talk) 17:38, 8 May 2023 (UTC)[reply]

That's funnny! I myself was surprised it was as high as 30% and can only conclude this is thanks to the w:WikiProject Women in Red. Don't forget there are several big issues with women that men don't have that makes data collection (and thus findability for Wikipedians wishing to create articles) very complex and difficult. Obviously the biggest issue is systemic bias by language (no info in English? article ain't happening). Women tend to travel less than men, due to family issues, making their chances of crossing language lines in news articles much less on the whole as a group. Secondly, women get married and change their name, clouding findability by genealogy methods. Thirdly, women hide their birth dates due to ageism in various fields, which again clouds findability for potential Wikipedians. The main issue that could contribute to a higher degree of article deletions for women and minorities is thus a lack of sources, but this does not address "ghost edits" or articles that were started but never happened. Collecting data on articles by Wikipedia project is definitely useful, but it gets interesting only when you drill down into things like occupation, notable works, and birth country, among other datapoints. Jane (talk) 07:04, 9 May 2023 (UTC)[reply]

I revisited that query I ran because I doubted the 30% and indeed the number is less than 30%. It occurred to me that my earlier query ran rather quickly and this was because I queried those born on 2000-01-01 and not after 2000-01-01. When I correct it I get timeouts, but this query (toggle manually for gender) for births between 2000 and 2010 will return 9705 females and 26672 males so 9705/(9705+26672)=27%, not 30%. This query runs towards the time limit for each gender, which is about right for English Wikipedia. Jane (talk) 09:43, 9 May 2023 (UTC)[reply]

Thanks for contributing some data! This paper did something very similar, just without the restriction to Wikipedia sitelinks (i.e. examined gender gaps on Wikidata instead of Wikipedia). They observed that the proportion of Wikidata items about women ranges between 0.2 and 0.25 for birth years 1950 to 1990 and has increased steadily since then, reaching 0.4178 for the 2000 birth year. (Also, they found that Wikidata editors are likely to over sample male-dominated professions such as American football and baseball, thus contributing to the general predominance of items representing men over items representing women. Our analysis that focused on a set of academic professions show that the gender distribution of Wikidata is no more biased than real world notability judgments in either coverage or quality. Obviously that doesn't speak directly to possible gender biases on Wikipedia instead of Wikidata, but it is still interesting e.g. because of similarities in the editor populations that people often like to draw quick conclusions from.)

Regards, HaeB (talk) 17:44, 8 May 2023 (UTC)[reply]

Without going into a whole list of pet peeves I have with the gender data on Wikidata, I will say it would be useful to query for the age of the subject at the time of item creation with the same moment recorded for each specific Wikipedia project with an actual article (minus any redirects) about the subject. If you could query that data and cross-reference it per Wikipedia language you could get some more specific information about gender bias on Wikipedia. My gut feeling for women in sports is that we have a large quantity of women thanks only to the wp:WikiProject Olympics in some Wikipedia languages. Currently I have no way to measure that. On the whole, I don't think studying AfD data is very useful for gender issues, but possibly it is for occupational issues (controversial (garage bands), new (vloggers) or individual-based (artists) and so forth). Jane (talk) 07:54, 10 May 2023 (UTC)[reply]

Thanks to XOR'easter, with editorial guidance from Tilman Bayer, for exposing the shortcomings of the abysmally bad paper by Mackenzie Lemieux, Rebecca Zhang and Francesca Tripodi about Wikipedia procedures for assessing the notability of scholars and researchers that was recently published in the journal Big data and society. I hope that we shall see a response by Lemieux, Zhang and Tripodi to the many allegations of inaccuracy and misrepresentation that exist in the paper. The editor of the journal Matthew Zook is invited to explain how this egregiously erroneous paper got through the journal’s peer review process. Xxanthippe (talk) 08:52, 8 May 2023 (UTC).[reply]

Great read. One minor thing I think might discourage people from sharing it outside Wikipedia is the use of the jargon "!voter" and "!vote". They're probably better replaced with "participant" and "comment" or the like. Nardog (talk) 09:13, 8 May 2023 (UTC)[reply]

Really good read by @XOR'easter and HaeB: - an excellent rebuttal of a terrible piece of academic writing. Hopefully Lemieux et al will read it and respond to just how so many flaws made it through. Though if I had to guess, they'll find a single line they can take out of context and use that to suggest the entire thing is clearly flawed. But we will see, and it doesn't undermine the quality of this piece. With thanks, Nosebagbear (talk) 09:51, 8 May 2023 (UTC)[reply]

Thanks! Just to clarify though, the text is all XOR'easter's (apart from the explanatory intro that quotes from the paper). On the other hand, it seems that some credit for identifying issues in the paper should also go to other participants in the discussion at the Women in Red talk page that had been started there at the beginning of April. (I myself only read the paper this past weekend, in the context of reviewing XOR'easter's Signpost submission, and subsequently contributed these observations about the authors' misinterpretation of p-values in one of their central quantitative conclusions.) Regards, HaeB (talk) 18:01, 8 May 2023 (UTC) (Tilman)[reply]

Thanks for the feedback! In the original version, I explained about deletion debates not being votes and what "!vote" means, but that got cut when bringing the column over to the Signpost. The original was even longer than this version and included more background for people coming to the topic without experience in Wikipedia behind-the-scenes stuff. XOR'easter (talk) 12:19, 8 May 2023 (UTC)[reply]

Excellent work. I would encourage you to submit the longer version for publication in a scholarly journal (perhaps in Big Data & Society itself). —Compassionate727 ^(T·C) 16:11, 8 May 2023 (UTC)[reply]

Over at the Women in Red project discussion page, we've been contemplating what to do next (and also going further into the statistics, which are ... puzzling: just how many biographies did they look at?). It's the end of the semester for me, so what with the usual obligations I haven't much spare time or energy. And writing about this isn't simple, on account of the tough balancing act: the problem of systemic bias is obviously important, but a specific paper about it can still be methodologically flawed beyond repair. Wikipedia's coverage of minority populations evidently isn't where it should be if we are to live up to our ideals, but a paper with that premise can still go so far astray from how things actually happen here that it enters the empyrean level of confident wrongness more typically associated with ChatGPT and men at cocktail parties. XOR'easter (talk) 16:32, 8 May 2023 (UTC)[reply]

I find it weird for a publication to present views of some importance when they can be manipulated. The Google search is more of a litmus test and never was a set guideline. The publication finding out should have been mentioned as a positive because we don't look at the info that SEO could easily sway. We look at more specific guidelines from GNG for people in their fields because it should make sense. – The Grid (talk) 13:22, 8 May 2023 (UTC)[reply]

This blithe and dismissive take on a well-established and important issue is unfortunate. It may be fair to criticise some of the methodology of their study, but the arguments in response to their criticisms presented above are specious. It would be far better to acknowledge that Wikipedia has well-established systemic biases and then we can figure out why that is the case. To refer to policy obfuscates from how that policy plays out in practice; and I think that most honest editors would willingly acknowledge that Wikipedia suffers from severe bias problems. Jack4576 (talk) 15:10, 8 May 2023 (UTC)[reply]

That's a lot to assume. People will quote essays, policies, and guidelines in AfD discussions to get the ball rolling on discussion. Note that essays and guidelines are not policies. Sometimes people point to WP:AFDOUTCOMES but always note citing any of the above with no explanation can be weighed lower than explanations made. The study seems to really not go into the meat of AfD discussions where discussion is always considered more important than people simply stating essays, policies, or guidelines. There's a tally but it's always been a very rough estimate. The criticism is valid. – The Grid (talk) 16:46, 8 May 2023 (UTC)[reply]

@Jack4576: this reasoning would seem to indicate that no paper should receive detailed criticism for its failings so long as it raises an overall issue of importance. Do you believe that specific criticisms raised in this article are incorrect? How does their being raised undermine the overall desire to combat bias? I also would like you to expand upon your indication of XOR's response being blithe, which means "lacking due thought or consideration". I can't see how this stands up, so seems to me more of an unsupported attack on the quality of their work. "Wikipedia suffers from severe bias problems" is certainly possible, but I would firmly dispute that AfD isn't policy driven. AfD is one of the most policy-backed fora on the entire site. While I'd appreciate a response now, I'd also be interested to see if your views remain unchanged after you've expanded your AfD experience set further, as you appear to be just starting into. Nosebagbear (talk) 16:58, 8 May 2023 (UTC)[reply]

In many cases, ostensibly policy driven. Which I believe is the whole point of this piece. It’s unsurprising that editors are keen to dismiss these critiques of hand. Jack4576 (talk) 23:19, 8 May 2023 (UTC)[reply]

it’s not a very strong critique to point to ostensible policies to dismiss an empirical study about the outcomes of said policy. No one claims Wikipedia’s policies are on their face biased; the core of the piece is what happens in practice with the implementation of said policies. Jack4576 (talk) 23:22, 8 May 2023 (UTC)[reply]

@Jack4576 But the core of the piece isn't about the actual practice. Otherwise it wouldn't spend large periods of time talking about non-policies/guidelines, it wouldn't need to engage in quote mining (the full quotes would give everything they needed), they wouldn't need to raise the idea that AfDs are biased for occurring and leave out they were closed for sexism in less than a couple of hours, and in particular they wouldn't need to make their own flagrantly incorrect statements of what various PAGs mean.

You could absolutely have a really good piece of research talking about whether the !votes people make execute the letter & spirit of the notability policies - but where the paper makes comparisons they do so to clearly incorrect statements of wikipedia policy, rendering them meaningless. I would also like your evidence of the scale of incorrect !votes due to biased interpretations that you say are occurring. Nosebagbear (talk) 12:05, 9 May 2023 (UTC)[reply]

This discussion is unproductive at this point. Obviously I don't have empirical evidence and am running off anecdotal experience like everyone else here. The article's conclusions ring true to me. Selective enforcement of policy can create bias, even if the policy itself is unbiased. This is obviously true and those who dispute that as occurring on Wikipedia just aren't paying close enough attention IMO. I suppose you need to wait for an empirical study to be convinced. You may be waiting a while. Jack4576 (talk) 12:41, 9 May 2023 (UTC)[reply]

I have looked at some of your AfD comments and you don't even provide Wikipedia policy for your reasons. It's really a blunder what you're going on about. – The Grid (talk) 12:55, 9 May 2023 (UTC)[reply]

I thought we were discussing this article. Feel free to chime in over on AfD if you have constructive feedback. Jack4576 (talk) 13:57, 9 May 2023 (UTC)[reply]

There's plenty of sources that discuss the problem of systemic bias on Wikipedia. But, this paper is worthless for determining its extent, nature, and causes, because its methodology is wrong. Most people would agree that there are issues of bias in, say, the criminal justice system, but an analysis paper on such bias that was based around, say, the magazines left in the police station waiting room rather than the briefings & trainings the police chief gave to the officers would be too nonsensical to be useful. SnowFire (talk) 19:51, 8 May 2023 (UTC)[reply]

worthless is overstating the case. Jack4576 (talk) 23:20, 8 May 2023 (UTC)[reply]

It really isn't. The errors described are catastrophic, along with a touch of nonsensical fake-math from people who don't know statistics. Your user page says you're interested in Australian law - imagine reading a paper with a conclusion you agree with, but the argument cites the 19th century Code Napoleon as if it were Australian law, and thinks that Ned Kelly was Prime Minister of Australia. Maybe the conclusion is correct, but the paper is still worthless. The authors clearly never ran their draft past an experienced Wikipedian. SnowFire (talk) 12:50, 9 May 2023 (UTC)[reply]

Personally, I would be shocked if there wasn't some degree of systemic bias in our handling of notability. I think there is absolutely a conversation to be had there. But we cannot reasonably have that conversation when the facts have been so badly muddied that essays are being described as guidelines and the search engine test is being held up as an example of How Notability Works. If your factual information is that poor, then just about the only thing you've told us is that there is systemic bias, somewhere, maybe, which I imagine most of us probably already knew. --N Y Kevin 04:05, 9 May 2023 (UTC)[reply]

And if we didn't know, we sure aren't going to be convinced by reading that. —Compassionate727 ^(T·C) 08:38, 9 May 2023 (UTC)[reply]

The selective quoting in the paper is so precisely selective in every single instance in order to omit actual statements of substance about notability from those being quoted that I can't help but think it was entirely purposeful by Lemieux et al. in order to farm quotes that present a viewpoint that supports the claims of the paper itself. Pretty disgraceful, if you ask me. Silver seren^C 16:46, 8 May 2023 (UTC)[reply]

Great review article, kudos to the contributors CactiStaccingCrane (talk) 01:02, 18 May 2023 (UTC)[reply]