Wikipedia:Wikipedia Signpost/2018-08-30/Recent research

Wehrmacht on Wikipedia, neural networks writing biographies: Readers prefer the AI's version 40% of the time – but it still suffers from hallucinations.

Recent research

Wehrmacht on Wikipedia, neural networks writing biographies

By Tilman Bayer and Bri

A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

The Battle for Wikipedia

Does Wikipedia lack true historical consensus on the actions of the Wehrmacht? Is our story skewed towards apologetic historiography like Lost Victories and the protesters against Wehrmachtsausstellung, shown here?

Reviewed by Bri

The "clean Wehrmacht" battle covered in the past three issues of The Signpost (May, June, July) is reviewed from a historian's perspective in The Journal of Slavic Military Studies. The title of the paper is an allusion to Lost Victories, today generally accepted as an unreliable and apologetic account of the actions of German forces during World War II. The author, David Stahel, who states that he is not a Wikipedia editor, examines the behind-the-scenes mechanisms and debates that result in article content, with the observation that these debates are not consistent with "consensus among serious historians" and "many people (and in my experience students) invest [Wikipedia] with a degree of objectivity and trust that, at least on topics related to the Wehrmacht, can at times be grossly misplaced...articles on the Wehrmacht (in English Wikipedia) might struggle to meet [the standard]". The author describes questionable arguments raised by several of the pro-Wehrmacht editors and concludes their writing "may in some instances reflect extremist views or romantic notions not grounded in the historiography".

Readers prefer summaries written by a neural network over those by Wikipedians 40% of the time – but it still suffers from hallucinations

Reviewed by Tilman Bayer

Several recent publications tackle the problem of taking machine-readable factual statements about a notable person, such as their date of birth from the Wikidata item about them, and creating a biographical summary in natural language.

A paper by three researchers from Australia reports on using an artificial intelligence approach for "the generation of one-sentence Wikipedia biographies from facts derived from Wikidata slot-value pairs". These are modeled after the first sentences of biographical Wikipedia articles, which, the authors argue, are of particular value because they form "clear and concise biographical summaries". The task of generating them involves making decisions about which of the facts to include (e.g. the date of birth or a political party that the subject is a member of), and arranging them into a natural language sentence. To achieve this in an automated fashion, the authors trained a recurrent neural network (RNN) implemented in TensorFlow on a corpus of several hundred thousand introductory sentences extracted from English Wikipedia articles about human, together the corresponding Wikidata entries. (Although not mentioned in the paper, such first sentences are the subject of a community guideline on the English Wikipedia, at least some aspects of which one might expect the neural network to reconstruct from the corpus.)

An example the algorithm's output compared to the Wikipedia original (excerpted from Table 5 in the paper):

Wikipedia original	robert charles cortner ( april 16 , 1927 may 19 , 1959 ) was an american automobile racing driver from redlands , california .
Algorithm variant "S2S"	bob cortner ( april 16 , 1927 may 19 , 2005 ) was an american professional boxer.
Algorithm variant "S2S+AE"	robert cortner ( april 16 , 1927 may 19 , 1959 ) was an american race-car driver .

The quality of the algorithm's output (in several variants) was evaluated against the actual human-written sentences from Wikipedia (as the "gold standard") with a standard automated test (BLEU), but also by human readers recruited from CrowdFlower. This "human preference evaluation suggests the model is nearly as good as the Wikipedia reference", with the consensus of the human raters even preferring the neural network's version 40% of the time. However, those of the algorithm's variants that are allowed to infer facts not directly stated in the Wikidata item can suffer from the problem of AI "hallucinations", e.g. the struck-out parts in the above example, claiming that Bob Cortner was a boxer instead of a race-car driver, and died in 2005 instead of 1959.

Apart from describing and evaluating the algorithm, the paper also provides some results about Wikipedia itself, e.g. showing which biographical facts are most frequently used by Wikipedia editors. Table 1 from the paper lists "the top fifteen slots across entities used for input, and the % of time the value is a substring in the entity’s first sentence" in the examined corpus:

Fact	Count	%
TITLE (name)	1,011,682	98
SEX OR GENDER	1,007,575	0
DATE OF BIRTH	817,942	88
OCCUPATION	720,080	67
CITIZENSHIP	663,707	52
DATE OF DEATH	346,168	86
PLACE OF BIRTH	298,374	25
EDUCATED AT	141,334	32
SPORTS TEAM	108,222	29
PLACE OF DEATH	107,188	17
POSITION HELD	87,656	75
PARICIPANT OF	77,795	23
POLITICAL PARTY	74,371	49
AWARD RECEIVED	67,930	44
SPORT	36,950	72

The paper's literature review mentions a 2016 paper titled "Neural Text Generation from Structured Data with Application to the Biography Domain" as "the closest work to ours with a similar task using Wikipedia infoboxes in place of Wikidata. They condition an attentional neural language model (NLM) on local and global properties of infobox tables [...] They use 723k sentences from Wikipedia articles with 403k lower-cased words mapping to 1,740 distinct facts".

While the authors of both papers commendably make at least some of their code and data available on GitHub (1, 2), they do not seem to have aimed to make their algorithms into a tool for generating text for use in Wikipedia itself – perhaps wisely so, as previous efforts in this direction have met with community opposition due to quality concerns (e.g. in the case of a paper we covered previously here: "Bot detects theatre play scripts on the web and writes Wikipedia articles about them").

In the third, most recent research effort, covered in several publications, another group of researchers likewise developed a method to automatically generate summaries of Wikipedia article topics via a neural network, based on structured data from Wikidata (and, in one variant, DBpedia).

They directly worked with community members from two small Wikipedias (Arabic and Esperanto) to evaluate "not only the quality of the generated text, but also the usefulness of our end-system to any underserved Wikipedia version", when extending the existing ArticlePlaceholder feature that is in use on some of these smaller Wikipedias. The result was that "that members of the targeted language communities rank our text close to the expected quality standards of Wikipedia, and are likely to consider the generated text as part of Wikipedia. Lastly, we found that the editors are likely to reuse a large portion of the generated summaries [when writing actual Wikipedia articles], thus emphasizing the usefulness of our approach to its intended audience."

Other recent publications

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions are always welcome for reviewing or summarizing newly published research.

Compiled by Tilman Bayer

The Wikipedia Adventure: Beloved but ineffective

A character in The Wikipedia Adventure

"The Wikipedia Adventure: Field Evaluation of an Interactive Tutorial for New Users"

From the accompanying blog post: "The system was a gamified tutorial for new Wikipedia editors. Working with the tutorial creators, we conducted both a survey of its users and a randomized field experiment testing its effectiveness in encouraging subsequent contributions. We found that although users loved it, it did not affect subsequent participation rates."

See also: research project page on Meta-wiki, podcast interview, podcast coverage, Wikimedia Research Showcase presentation

Told you so: Hindsight bias in Wikipedia articles about events

Two papers by the same team of researchers explore this topic for Wikipedia editors and readers, respectively:

"Biases in the production and reception of collective knowledge: the case of hindsight bias in Wikipedia"

From the paper:

Study 1: This study investigated whether events in Wikipedia articles are represented as more likely in retrospect. For a total of 33 events, we retrieved article versions from the German Wikipedia that existed prior to the event (foresight) or after the event had happened (hindsight) and assessed indicators of hindsight bias in those articles [...] we determined the number of words of the categories "cause" (containing words such as "hence"), "certainty" (e.g., "always"), tentativeness (e.g., "maybe"), "insight" (e.g., "consider"), and "discrepancy" (e.g., "should"), because the hindsight perspective is assumed to be the result of successful causal modeling [...] There was an increase in the proportion of hindsight-related words across article versions. [...] We investigated whether there is evidence for hindsight distortions in Wikipedia articles or whether Wikipedia’s guidelines effectively prevent hindsight bias to occur. Our study provides empirical evidence for both.

"Cultural Interpretations of Global Information? Hindsight Bias after Reading Wikipedia Articles across Cultures"

From the abstract: "We report two studies with Wikipedia articles and samples from different cultures (Study 1: Germany, Singapore, USA, Vietnam, Japan, Sweden, N = 446; Study 2: USA, Vietnam, N = 144). Participants read one of two article versions (foresight and hindsight) about the Fukushima Nuclear Plant and estimated the likelihood, inevitability, and foreseeability of the nuclear disaster. Reading the hindsight article increased individuals' hindsight bias independently of analytic or holistic thinking style. "

"WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval"

From the abstract: "...we introduce a new Wikipedia based collection specific for non-factoid answer passage retrieval containing thousands of questions with annotated answers and show benchmark results on a variety of state of the art neural architectures and retrieval models."

"Analysis of Wikipedia-based Corpora for Question Answering"

From the abstract: "This paper gives comprehensive analyses of corpora based on Wikipedia for several tasks in question answering. Four recent corpora are collected, WikiQA, SelQA, SQuAD, and InfoQA, and first analyzed intrinsically by contextual similarities, question types, and answer categories. These corpora are then analyzed extrinsically by three question answering tasks, answer retrieval, selection, and triggering."

"Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia"

From the abstract: "We study the task of generating from Wikipedia articles question-answer pairs that cover content beyond a single sentence. We propose a neural network approach that incorporates coreference knowledge via a novel gating mechanism. [...] We apply our system [...] to the 10,000 top-ranking Wikipedia articles and create a corpus of over one million question-answer pairs."

Asking Wikidata questions in natural language

From the abstract: "We first introduce a new approach for translating natural language questions to SPARQL queries. It is able to query several KBs [knowledge bases] simultaneously, in different languages, and can easily be ported to other KBs and languages. In our evaluation, the impact of our approach is proven using 5 different well-known and large KBs: Wikidata, DBpedia, MusicBrainz, DBLP and Freebase as well as 5 different languages namely English, German, French, Italian and Spanish." Online demo: https://wdaqua-frontend.univ-st-etienne.fr/

References

← Previous "Recent research"

Next "Recent research" →

In this issue

30 August 2018 (all comments)

From the editor

Interview

News and notes

In the media

Discussion report

Featured content

Special report

Traffic report

Technology report

Gallery

Recent research

Humour

Essay

From the archives

+ Add a comment

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try .

• From the journal: "many people (and in my experience students) invest [Wikipedia] with a degree of objectivity and trust". I think that's quite far from reality. Even younger kids tend to be rather level-headed about Wikipedia and teachers will tell how to treat Wikipedia as a source in essays. Wikipedia, and the fact that anyone can edit it, isn't anything new anymore. --Pudeo (talk) 07:47, 30 August 2018 (UTC)

  @Pudeo: Trust varies according to subject area and the depth of information being looked for, which the author alluded to. People might be quite happy looking up a small fact on Wikipedia to refresh their memory without taking everything they read there at face value. The statement doesn't seem particularly controversial given how much Wikipedia is used, and the author is not saying that Wikipedia is seen an unimpeachable authority, but that it is seen to have some objectivity. There is also the 2014 YouGov poll which found that in Britain contributors to Wikipedia entries were more trusted than journalists. Richard Nevell (talk) 10:09, 31 August 2018 (UTC)

  Thanks for that very interesting link to the 2014 YouGov poll, Richard Nevell. I'm not that surprised, though it is curious how the editors of an encyclopedia that is necessarily a derivative of the information provided by those journalists can be considered more trustworthy than them. That derivation should reasonably mean that Wikipedia's trustworthiness cap is the value of the journalists' trustworthiness. The fact that trust of Wikipedians is higher than journalists either says an immense lot about Wikipedians (and journalists) or at least something about the poll participants—or both. I'd love to see a 2018–19 update to it. —Nøkkenbuer (talk • contribs) 01:22, 1 September 2018 (UTC)

• "pro Wehrmacht editors": I have never been pro Wehrmacht, nor am I even interested in the Germany military. I did, however, try to offer an opinion about Wikipedia policy on sourcing (whether I'm right or wrong is another question) on a talk page to try to resolve a dispute (obviously unsuccessfully), and for my troubles got quoted out of context without any right of reply before publication. But anyway. Regards, AustralianRupert (talk) 09:47, 1 September 2018 (UTC)

• Agree "pro-Wehrmacht" is uncalled for, and is not supported by materials presented in the case which made no such finding regarding any editor. Bri, would you consider removing this particular phrase, or at least indicating a direct attribution to the offwiki author so the term does not appear to be in Wikipedia's (or The Signpost's) voice? -- Euryalus (talk) 10:38, 1 September 2018 (UTC) Probably needless disclaimer - the request that this particular phraseology in the article be reconsidered is my personal opinion and not on behalf of Arbcom. -- Euryalus (talk) 11:20, 1 September 2018 (UTC)

• The research doesn't lend itself well to succinct phrasing which is needed in this review. The nearest relevant phrase I can find to the passage in question is "too many of the editors are clearly keen to see articles that reflect their own cherished, or at least uncritical, view of the Wehrmacht and its collaborators." I think "pro-Wehrmacht" is a fair and accurate condensation of that thought. I think it will remain as written. ☆ Bri (talk) 13:16, 1 September 2018 (UTC)

• Thanks for the reply. Stahel certainly accuses some editors of being "pro-Wehrmacht," without offering much evidence. Charitably, perhaps that's a consequence of the need for brevity in his piece. I suppose my question is whether the current Signpost wording implies that the Signpost accepts Stahel's accusation as fact, or simply notes that Stahel said it.-- Euryalus (talk) 13:41, 1 September 2018 (UTC)

• It's already prefaced with "The author describes...". Are you suggesting further qualification? ☆ Bri (talk) 15:10, 1 September 2018 (UTC)

• I agree that this wording is unfair to the editors here. Stahel actually uses the term "pro-Wehrmacht", but only in relation to "the notoriously pro-Wehrmacht J.J. Fedorowicz Publishing" company, not Wikipedia editors. He uses much more convoluted wording to describe the Wikipedia editors he focuses on, but the gist of it is that he sees their views of the German military as being outdated rather than actually advocating on its behalf as the term "pro" implies (e.g. "The problem is as much about what is written as what is left out and sometimes what is removed by editors acting, consciously or unconsciously, to preserve the myth of a ‘clean Wehrmacht’"). More broadly, the term doesn't reflect the findings of the arbitration case, or my years of experience with the editors in question. It should be removed. Nick-D (talk) 08:17, 6 September 2018 (UTC)

• Belatedly, agree with Nick-D. The issue is the article's use of "pro-Wehrmacht," in The Signpost's voice, to describe editors named in the referenced article. Stahel certainly implies that accusation, though he doesn't say it explicitly. The Arbcom case outcome goes nowhere near such a claim. Of course it's up to the editorial group on whether to keep or remove the sentence, though as a passing reader of the Signpost I'd urge removal. But if the editorial decision is to keep this sentence, perhaps it could be amended with words like "editors who Stahel implies are pro-Wehrmacht" to make absolutely clear it is merely reporting someone else's views. -- Euryalus (talk) 08:58, 6 September 2018 (UTC)

FWIW, I would think it should be worth mentioning that a passage Stahel complains about having been cut, at the bottom of page 398, cited his own work as a source? He may have less disinterested motives in writing this than it seems. Daniel Case (talk) 20:51, 2 September 2018 (UTC)

• In the Stahel article (https://www.tandfonline.com/doi/pdf/10.1080/13518046.2018.1487198), we can only find a non-critical timeline of the recent cup-of-tea-storm that occurred at en:wp. The key question was if Wikipedia has been used or not as a loud speaker to advertise a "pro-Nazi" point of view. The fact that all this tempest ended by Wikipedia:Arbitration/Requests/Case/German_war_effort#Findings_of_fact is not mentioned by Stahel. In other words, this was only another tempest about another pokemon series: the so-called German Army "aces". Praising a quidam for "successfully conducting a car over the distance from Los Angeles to El Paso" is surely an access of fan-cruft. Even the porn-star wp-articles aren't going that far. But this can not be brushed as Nazi propaganda, whatever source is used to backup such an irrelevant factoid.

Moreover, it is largely surprising that, after being so vocal about historical context, there is not even the smallest allusion to the fact that large white-washing campaigns have been undertaken in the past, to allow the recycling of the defeated mass-murderers into murderers with our God on their side. MacArthur protecting Hirohito, Churchill protecting Kesselring, and so on were not isolated facts... but describing this situation as orchestrated by Wikipedia is too large a brush... and slightly anachronistic.

Stahel concludes by saying: "the best advice for students is not to risk Russian roulette on the Internet and instead seek peer-viewed literature from the library"... and just the next line, the same students are advised that "David Stahel has written four books about the Wehrmacht’s operations on the eastern front with Cambridge University Press". What a marvelous coincidence ! Pldx1 (talk) 08:44, 6 September 2018 (UTC)

The Signpost needs your help putting together the next issue.

Home

About

Archives

Newsroom

Subscribe

Suggestions