Wikipedia talk:Wikipedia Signpost/2023-10-03/Recent research

Discuss this story

FlaggedRevs

There seem to be some errors in The Risks, Benefits, and Consequences of Prepublication Moderation: Evidence from 17 Wikipedia Language Editions (https://arxiv.org/abs/2202.05548) papers assumptions on how FlaggedRevs works. For example:

... FlaggedRevs is a prepublication content moderation system in that it will display the most recent “flagged” revision of any page for which FlaggedRevs is enabled instead of the most recent revision in general... - This is depending on wiki where user is. For example German Wikipedia uses pre-moderation, and as different setup example is Finnish Wikipedia which uses post-moderation.
... Although editors without accounts remain untrusted indefinitely, editors with accounts are automatically promoted to trusted status when they clear certain thresholds determined by each language community.... - This also depends on wiki. For example dewiki promotes automatically, but huwiki has rather small number manually selected trusted reviewers. Fiwiki has large number manually selected trusted reviewers.

--Zache (talk) 09:11, 4 October 2023 (UTC)[reply]

In Russian Wikipedia, as well as in Russian Wikinews, FlaggedRevs is a disaster. You say Germans are guilty in that? --ssr (talk) 06:09, 27 October 2023 (UTC)[reply]

Wikidata

Wikidata as Semantic Infrastructure: Knowledge Representation, Data Labor, and Truth in a More-Than-Technical Project.: The article has a lot of critiques of Wikidata, some interesting, some less relevant in my opinion. Some of them were a big part of the discussions around adopting Wikidata on frwiki, interestingly, so might be a little bit out of date. The copyright issues, for example, was and is still I guess a show stopper for some. Interestingly I think this point is significantly weakened with the emergence of the LLMs like ChatGPT as a major products of the Gafams : they don’t really care about the copyright issues or the attribution of datas they learn from, and they really can’t track back the generated factoids origin or reliably source anything … Without any restraint to date from copyright laws. Copyright does not seem to slow automated fact extraction from any public source in any way … (for the rest of the arguments covered for or against Wikidata on wikipedias, this has already been extensively discussed and argued on the projects, I won’t return on this because I don’t really feel like it); I don’t quite understand the upper ontology part, as I think Wikidata as is is difficult to really use as an upper ontology because of the lack of formalisation. There is no reasoning rules in Wikidata to capture some of the meaning, and this is a crucial part of how an upper ontology is supposed to be used. The upper tree is indeed, or as long been as there is ongoing effort to clear this up, globally a hard to manage mess. But I doubt anyone actually used in a meaningful way the classes they show up like bad, or actually used Wikidata as an upper ontology. I’d like this claim of the article to be given more substance or proof.; It’s a critic I would actually make for the whole article, this is a discussion of the critiques but there is not really a lot of datas to give substance, this is mostly qualitative, and as such arguable. TomT0m (talk) 11:38, 4 October 2023 (UTC)[reply]

ChatGPT v. Wikipedia

The study authors comment on prose quality. I happened to ask ChatGPT yesterday to explain what government shutdowns in the U.S. are and what effects they have. I got the following answer:

Government shutdowns in the United States occur when Congress fails to pass a budget or a continuing resolution to fund the government. This can happen due to disagreements between political parties or issues related to specific policies. The shutdown results in the temporary suspension of non-essential government services and the furlough of federal employees.

During a government shutdown, essential services such as national security, law enforcement, and emergency activities continue to operate. However, many government functions, agencies, and national parks may be affected. Federal employees who are deemed non-essential may be furloughed, meaning they are placed on temporary unpaid leave.

The economic impact can be significant, affecting various sectors, including tourism, government contractors, and the overall economy. Additionally, government shutdowns can lead to a loss of public confidence in the government's ability to function effectively.

It's worth noting that my information is up to September 2021, and I recommend checking the latest sources for updates on this topic.

I then compared that to the lead of Government shutdowns in the United States:

In the United States, government shutdowns occur when funding legislation required to finance the federal government is not enacted before the next fiscal year begins. In a shutdown, the federal government curtails agency activities and services, ceases non-essential operations, furloughs non-essential workers, and retains only essential employees in departments that protect human life or property. Shutdowns can also disrupt state, territorial, and local levels of government.

Funding gaps began to lead to shutdowns in 1980, when Attorney General Benjamin Civiletti issued a legal opinion requiring it. This opinion was not consistently adhered to through the 1980s, but since 1990 all funding gaps lasting longer than a few hours have led to a shutdown. As of September 2023, ten funding shutdowns have led to federal employees being furloughed.

The most significant shutdowns have included the 21-day shutdown of 1995–1996, during the Bill Clinton administration, over opposition to major spending cuts; the 16-day shutdown in 2013, during the Barack Obama administration, caused by a dispute over implementation of the Affordable Care Act (ACA); and the longest, the 35-day shutdown of 2018–2019, during the Donald Trump administration, caused by a dispute over funding an expansion of barriers on the U.S.–Mexico border.

Shutdowns disrupt government services and programs; they close national parks and institutions. They reduce government revenue because fees are lost while at least some furloughed employees receive back pay. They reduce economic growth. During the 2013 shutdown, Standard & Poor's, the financial ratings agency, said on October 16 that the shutdown had "to date taken $24 billion out of the economy", and "shaved at least 0.6 percent off annualized fourth-quarter 2013 GDP growth".

Personally I found ChatGPT's output a lot more readable than the Wikipedia lead – it is just better written. The English Wikipedia text often required me to go back and read the sentence again.

Take the first sentence: In the United States, government shutdowns occur when funding legislation ... At first I parsed "when funding legislation" as an indication of when shutdowns occur (i.e. "when you are funding legislation"). I needed to read on to realise that this wasn't where the sentence was going.

Next, Wikipedia uses the rather technical expression "when funding legislation ... is not enacted" (which is also passive voice) where ChatGPT uses the much easier-to-understand "when Congress fails to pass a budget" (active voice).

Where ChatGPT speaks of a "temporary suspension of non-essential government services", Wikipedia says the federal government "curtails agency activities and services, ceases non-essential operations", etc. I find the ChatGPT phrase easier to understand and faster to read while providing much the same information as the quoted Wikipedia passage (a point the study authors commented on specifically).

The Wikipedia sentence Funding gaps began to lead to shutdowns in 1980, when Attorney General Benjamin Civiletti issued a legal opinion requiring it. leaves me wondering even now what the word "it" at the end of the sentence is meant to refer to.

I suspect our sentence construction and word use are not helping us win friends. It's one thing when we are the only service available; it's another when there is a new kid on the block. Andreas JN 466 13:56, 4 October 2023 (UTC)[reply]

I agree that ChatGPT produces better prose than most humans do (myself included), and as far as reliability is concerned, I'd honestly trust it to give me a general overview on any non-controversial topic, no matter how obscure (I tested it earlier on "the Black sermonic tradition of whooping" and it passed with flying colours). I think where it fails and starts to hallucinate is when you ask it to drill down into the detail. This is Wikipedia's strength – we may not be good at summarizing complex topics, but we're very good at detail.

So I don't think it has to be "machines vs. Wikipedians". I used ChatGPT this morning to write a lead for the article Khlysts. As leads go, it's maybe not perfect, but it's decent. But when I ask the bot for more in-depth information, like "Who founded the Khlysts?", it starts to get vague, and if I press it for specific answers, that's when it starts lying rather than admit it doesn't know. Chatbots have got a long way to go before they can write entire articles (at least without a lot of coaching and cross-checking, at which point you're not really saving any time), and I personally don't believe that AI will ever reach that level. But I can easily foresee a not-too-distant-future in which Wikipedia has an integrated LLM interface that allows editors to auto-generate summary-style content, or perhaps provides it to readers on the fly via a "Summarize this article/section" button.

In my view, the alarm of the study authors is misplaced. They seem to agree that the AI-generated content they used in the study is, in fact, trustworthy and competent, so there's no reason to be "unsettled" that people rate it as such. I suspect the participants would give a different rating to the vague, generic answers that ChatGPT gives you when you try to go beyond a summary. I don't think there's any competition here – when people want pedantic, granular detail, they'll always know where to find it. Sojourner in the earth (talk) 21:52, 4 October 2023 (UTC)[reply]

The fact that ChatGPT doesn't know when it doesn't know something means it's "intelligent" only in a restricted sense. To go on and on about a topic, all the while presenting itself as an "intelligence", also does not seem ethical. CurryCity (talk) 08:29, 5 October 2023 (UTC)[reply]

Quoted from above, "I'd honestly trust it to give me a general overview on any non-controversial topic, no matter how obscure" — that's extremely problematic. Recently I read about someone who is a critic with a dim view of a certain thing, a fact that is well known in his field (albeit not by the general public), that when he asked ChatGPT who he is, it smoothly and confidently claimed that he is a proponent of it. The devil is in details of the black box of "where it fails and starts to hallucinate is when you ask it to drill down into the detail" — but clearly the upshot is that you absolutely cannot trust GPT not to mix any bits of plausible-sounding-but-definitely-incorrect hallucination into any kind of "explanation" that it gives you about what something or someone is or isn't, when that something or someone is any more obscure than extremely widely known and understood. When I read in this Signpost article the sentence, " […] even if the participants know that the texts are from ChatGPT, they consider them to be as credible as human-generated and curated texts [from Wikipedia]," the first thought that popped into my head is, "Well, that's it, we're boned." Duck and cover and kiss our present standard of living goodbye, because some bad shit will be happening if that shit continues very long in its current configuration. You know how McNamara said that "the indefinite combination of human fallibility and nuclear weapons will destroy nations"? Like it doesn't even matter if the annualized risk is somewhat low, if you run the runtime for enough years? That's most likely what's going to happen with the combination of human [epistemologic] fallibility [incompetence] plus LLMs, unless the "indefinite" part can be massively shortened in duration by chaining today's type of LLMs in series with some kind of downstream rapid-acting bullshit-detector software that can act like a filter. If that shit doesn't get invented within the next N years, where time N is pretty fucking short, then there is going to come a time in the future where some Homer Simpson who works at a nuclear plant is going to ask an LLM how to run the reactor properly and the LLM is going to reply with some super-smoothly-worded and speciously/superficially-plausible-but-yet-idiotically-wrong answer, and there's going to be another Chernobyl-type steam explosion of a reactor, because even though the plant has a strict rule against asking any LLMs for any cheatsheet help, Homer is going to carry his smartphone into the bathroom and ask it while inside the toilet stall because he thinks he's getting away with a clever clandestine lifehack and sticking it to The Man. Or some librarian is going to ask an LLM which library books to ban for age-inappropriate content and it's totally going to confabulate a bunch of imaginary but plausible-sounding horseshit and get books banned for containing things that they don't even contain at all (that one already happened, by the way). Or a lawyer is going to ask an LLM to write a legal argument and cite a bunch of 'definitely real and please not fake' references that support it, and he's going to get fired because those smoothly cited plausible-sounding references are all made-up imaginary horseshit anyway (that one already happened, by the way). "Oh, no, don't be silly, Quercus, that reactor-explosion thing's not gonna happen." Yeah. We all sure better just hope not, right? Running on hope and faith but no guardrails at all, what could go wrong, right? This is why the likes of Sam Altman are begging governments to regulate their industry (albeit begging with crocodile tears in their eyes while they shovel wads of money into their bank accounts). Also, just to clear up a misconception implied on this page, among some of these other comments — regarding the notion that, "Well, how can you blame people, because a supposed-to-be-nonfiction answer that's factually wrong but super-smoothly worded and easy to read is preferable to one that's factually correct but somewhat clunkily worded." Jesus, Lord help us with the stupidity. If you have to ask what's wrong with that notion, then goto Homer's steam explosion above. Quercus solaris (talk) 23:37, 6 October 2023 (UTC)[reply]

Thing is, there are plenty of Wikipedia articles that also contain "plausible-sounding-but-definitely-incorrect" information, so for LLMs and Wikipedia to be rated as equally credible sounds about right. Of course I wouldn't use either as a source if I were writing my own article, but for satisfaction of personal curiosity, sure. As for the rest of your comment, I'm not convinced that "human fallibility plus LLMs" is a bigger danger to society than, say, human fallibility plus the invention of the printing press, or human fallibility plus firelighters. Pretty much anything that benefits humanity can be misused, and every Next Big Thing gets people prophesying the imminent collapse of civilization, but somehow we keep trucking along. Sojourner in the earth (talk) 05:40, 7 October 2023 (UTC)[reply]

In this case success and failure are asymmetric. It takes only one bad enough catastrophe once to wipe out civilisation, if not humanity. Even ones that are "too small" to destroy the rest of the biosphere or planet with us. CurryCity (talk) 06:33, 10 October 2023 (UTC)[reply]

Even if ChatGPT or its successor becomes the predominant internet search tool, that doesn't mean Wikipedia will be obsolete. It likely means that Wikipedia will go back to its theoretical origin as a reference work rather than the internet search tool many readers use it as. Thebiguglyalien (talk) 16:11, 4 October 2023 (UTC)[reply]

Ah, the rise of AI. I've used it to get ideas for small projects in the past, but people prefer LLMs over Wikipedia? That's, just... sad. The Master of Hedgehogs is back again! 22:09, 4 October 2023 (UTC)[reply]

It's quite understandable because articles often have a ponderous, pedantic style. As a fresh example, consider the current FA, ministerial by-election. Its first sentence is
From 1708 to 1926, members of parliament (MPs) of the House of Commons of Great Britain (and later the United Kingdom) automatically vacated their seats when made ministers in government and had to successfully contest a by-election in order to rejoin the House; such ministerial by-elections were imported into the constitutions of several colonies of the British Empire, where they were likewise all abolished by the mid-20th century.
When this was a candidate, a reviewer observed that
... you have a tendency to write very long sentences, exemplified by the lead sentence, which is 67 words long. A quick google search tells me 25 words is the optimal sentence length (and 30 the maximum), but here you can routinely see double that.
By my count, this sentence now has 68 words so it's only getting longer!

Andrew🐉(talk) 11:47, 5 October 2023 (UTC)[reply]

I really hope we can get a guideline on readability. My young sister-in-law complains that we adults "Explain things like Wikipedia", rather than in a way that a teenager can understand. I started to draft readability guideline once, but never found time to get it past the first couple of sentences. —Femke 🐦 (talk) 19:59, 6 October 2023 (UTC)[reply]

There are plenty of guidelines such as WP:NOTJARGON, WP:NOTJOURNAL and MOS:LEAD. The challenge is getting them followed. There are also plenty of metrics for this such as the automated readability index. We just need a bot or tool to assess all our articles and so highlight cases in need of attention. Andrew🐉(talk) 22:46, 6 October 2023 (UTC)[reply]

What is missing from these guidelines is a) the need to make non-technical text easy to understand b) more practical advice or what sentence and paragraph lengths make text difficult. I've been playing with the idea of creating a contest, similar to WP:the core contest, to tackle lead difficulty. Not sure yet what the best set-up is, as misguided attempts at simplifying can be quite bad. —Femke 🐦 (talk) 06:50, 7 October 2023 (UTC)[reply]

You have your work cut out for you as TFA is now featuring another good example: Affine symmetric group. ChatGPT struggles with the topic too. I reckon that it's possible to do better but it would be a thankless task currently. Andrew🐉(talk) 08:55, 8 October 2023 (UTC)[reply]

@Femke and Andrew Davidson: For what it's worth, you can ask ChatGPT or DeepL (which has a beta feature for this) to rewrite texts to make them easier to understand. Fed the Wikipedia lead section of Government shutdowns in the United States mentioned above and asked to improve its readability, ChatGPT produced the following:

In the U.S., government shutdowns happen when the funding needed to run the federal government isn't approved before the new fiscal year begins. During a shutdown, the government cuts back on agency activities and services, stops non-essential operations, puts non-essential workers on leave, and keeps only essential employees in departments that protect people or property. Shutdowns can also cause disruptions at the state, territorial, and local government levels.

Funding gaps started causing shutdowns in 1980, following a legal opinion from Attorney General Benjamin Civiletti. While this opinion wasn't consistently followed in the 1980s, since 1990, any funding gap lasting more than a few hours has led to a shutdown. As of September 2023, ten funding shutdowns have resulted in federal employees being placed on leave.

Major shutdowns include the 21-day shutdown in 1995–1996, during the Bill Clinton administration, due to disagreements over significant spending cuts; the 16-day shutdown in 2013, during the Barack Obama administration, caused by a dispute over implementing the Affordable Care Act (ACA); and the longest one, the 35-day shutdown in 2018–2019, during the Donald Trump administration, triggered by a disagreement over funding for expanding barriers on the U.S.–Mexico border.

Shutdowns disrupt government services and programs, leading to the closure of national parks and institutions. They also result in reduced government revenue as fees are lost, even though some furloughed employees eventually receive back pay. Furthermore, they negatively impact economic growth. For instance, during the 2013 shutdown, financial ratings agency Standard & Poor's reported on October 16 that the shutdown had already taken "$24 billion out of the economy" and "reduced annualized fourth-quarter 2013 GDP growth by at least 0.6 percent."

I wouldn't endorse every change, but would argue that the overall result is indeed improved readability. Andreas JN 466 11:51, 8 October 2023 (UTC)[reply]