Wikipedia:Wikipedia Signpost/2023-01-01/Technology report
Wikimedia Foundation's Abstract Wikipedia project "at substantial risk of failure"
Could Abstract Wikipedia fail?
In 2020, the Wikimedia Foundation began working on Abstract Wikipedia, which is envisaged to become the first new Wikimedia project since Wikidata's launch in 2012, accompanied and supported by the separate Wikifunctions project. Abstract Wikipedia is "a conceptual extension of Wikidata", where language-independent structured information is rendered in an automated way as human-readable text in a multitude of languages, with the hope that this will vastly increase access to Wikipedia information in hitherto underserved languages. Both Abstract Wikipedia and Wikifunctions are the brainchild of longtime Wikimedian Denny Vrandečić, who also started and led the Wikidata project at Wikimedia Deutschland before becoming a Google employee in 2013, where he began to develop these ideas before joining the Wikimedia Foundation staff in 2020 to lead their implementation.
An evaluation published earlier this month calls the project's future into question:
"This is a sympathetic critique of the technical plan for Abstract Wikipedia. We (the authors) are writing this at the conclusion of a six-month Google.org Fellowship, during which we were embedded with the Abstract Wikipedia team, and assisted with the development of the project. While we firmly believe in the vision of Abstract Wikipedia, we have serious concerns about the design and approach of the project, and think that the project faces a substantial risk." [...]
"We find [Abstract Wikipedia's] vision strongly compelling, and we believe that the project, while ambitious, is achievable. However, we think that the current effort (2020–present) to develop Abstract Wikipedia at the Wikimedia Foundation is at substantial risk of failure, because we have major concerns about the soundness of the technical plan. The core problem is the decision to make Abstract Wikipedia depend on Wikifunctions, a new programming language and runtime environment, invented by the Abstract Wikipedia team, with design goals that exceed the scope of Abstract Wikipedia itself, and architectural issues that are incompatible with the standards of correctness, performance, and usability that Abstract Wikipedia requires."
That Fellowship was part of a program by Google.org (the philanthropy organization of the for-profit company Google) that enables Google employees to do pro-bono work in support of non-profit causes. The Fellow team's tech lead was Ori Livneh, himself a longtime Wikipedian and former software engineer at the Wikimedia Foundation (2012–2016), where he founded and led the Performance Team before joining Google. The other three Google Fellows who authored the evaluation are Ariel Gutman (holder of a PhD in linguistics and author of a book titled "Attributive constructions in North-Eastern Neo-Aramaic", who also published a separate "goodbye letter" summarizing his work during the Fellowship), Ali Assaf, and Mary Yang.
The evaluation examines a long list of issues in detail, and ends with a set of recommendations centered around the conclusion that –
"Abstract Wikipedia should be decoupled from Wikifunctions. The current tight coupling of the two projects together has a multiplicative effect on risk and substantially increases the risk of failure."
Among other things, the Fellows caution the Foundation to not "invent a new programming language. The cost of developing the function composition language to the required standard of stability, performance, and correctness is large ..." They propose that –
- "Wikifunctions should extend, augment, and refine the existing programming facilities in MediaWiki. The initial version should be a central wiki for common Lua code", Lua being "an easy-to-learn and general purpose programming language, originally developed in Brazil" that is already widely used on Wikimedia projects, with the added benefit of satisfying "a long-standing community request" (for a "Central repository for gadgets, templates and Lua modules", which had been the third most popular proposal in the 2015 Community Wishlist Survey).
Regarding Abstract Wikipedia, the recommendations likewise center on limiting complexity and aiming to build on existing open-source solutions if possible, in particular for the NLG (natural language generation) part responsible for converting the information expressed in the project's language-independent formalism into a human-readable statement in a particular language:
- Rather than present to users a general-purpose computation system and programming environment, provide an environment specifically dedicated to authoring abstract content, grammars, and NLG renderers in a constrained formalism.
- Converge on a single, coherent approach to NLG.
- If possible, adopt an extant NLG system and build on it."
The Foundation's answer
A response authored by eight Foundation staff members from the Abstract Wikipedia team (published simultaneously with the Fellows' evaluation) rejects these recommendations. They begin by acknowledging that although "Wikidata went through a number of very public iterations, and faced literally years of criticism from Wikimedia communities and from academic researchers[, the] plan for Abstract Wikipedia had not faced the same level of public development and discussion. [...] Barely anyone outside of the development team itself has dived into the Abstract Wikipedia and Wikifunctions proposal as deeply as the authors of this evaluation."
However, Vrandečić's team then goes on to reject the evaluation's core recommendations, presenting the expansive scope of Wikifunctions as a universal repository of general-purpose functions a done deal mandated by the Board (the Wikimedia Foundation's top decision-making authority), and accusing the Google Fellows of "fallacies" rooted in "misconception":
The Foundation’s Board mandate they issued to us in May 2020 was to build the Wikifunctions new wiki platform (then provisionally called Wikilambda) and the Abstract Wikipedia project. This was based on the presentation given to them at that meeting (and pre-reading), and publicly documented on Meta. That documentation at the time very explicitly called out as “a new Wikimedia project that allows to create and maintain code” and that the contents would be “a catalog of all kind[s] of functions”, on top of which there would “also” (our emphasis) be code for supporting Abstract Wikipedia.
The evaluation document starts out from this claim – that Wikifunctions is incidental to Abstract Wikipedia, and a mere implementation detail. The idea that Wikifunctions will operate as a general platform was always part of the plan by the Abstract Wikipedia team.
This key point of divergence sets up much of the rest of this document [i.e. the evaluation] for fallacies and false comparisons, as they are firmly rooted in, and indeed make a lot of sense within, the reality posed by this initial framing misconception."
(The team doesn't elaborate on why the Foundation's trustees shouldn't be able to amend that May 2020 mandate if, two and a half years later, its expansive scope does indeed risk causing the entire project to fail.)
The evaluation report and the WMF's response are both lengthy (at over 6,000 and over 10,000 words, respectively), replete with technical and linguistic arguments and examples that are difficult to summarize here in full. Interested readers are encouraged to read both documents in their entirety. Nevertheless, below we attempt to highlight and explain a few key points made by each side, and to illuminate the underlying principal tensions about decisions that are likely to shape this important effort of the Wikimedia movement for decades to come.
What is the scope of the new "Wikipedia of functions"?
In an April 2020 article for the Signpost (published a few weeks before the WMF board approved his proposal), Vrandečić explained the concept of Abstract Wikipedia and a "wiki for functions" using an example describing political happenings involving San Francisco mayor London Breed:
"Instead of saying "in order to deny her the advantage of the incumbent, the board votes in January 2018 to replace her with Mark Farrell as interim mayor until the special elections", imagine we say something more abstract such as
elect(elector: Board of Supervisors, electee: Mark Farrell, position: Mayor of San Francisco, reason: deny(advantage of incumbency, London Breed))– and even more, all of these would be language-independent identifiers, so that thing would actually look more likeQ40231(Q3658756, Q6767574, Q1343202(Q6015536, Q6669880)).[...] We still need to translate [this] abstract content to natural language. So we would need to know that the
electconstructor mentioned above takes the three parameters in the example, and that we need to make a template such as{elector} elected {electee} to {position} in order to {reason}(something that looks much easier in this example than it is for most other cases). And since the creation of such translators has to be made for every supported language, we need to have a place to create such translators so that a community can do it.For this I propose a new Wikimedia project [...] to create, maintain, manage, catalog, and evaluate a new form of knowledge assets: functions. Functions are algorithms, pieces of code, that translate input into output in a determined and repeatable way. A simple function, such as the square function, could take the number 5 and return 25. The length function could take a string such as "Wikilambda" and return the number 10. Another function could translate a date in the Gregorian calendar to a date in the Julian calendar. And yet another could translate inches to centimeters. Finally, one other function, more complex than any of those examples, could take an abstract content such as
Q40231(Q3658756, Q6767574, Q1343202(Q6015536, Q6669880))and a language code, and give back the text "In order to deny London Breed the incumbency advantage, the Board of Supervisors elected Mark Farrell Mayor of San Francisco." Or, for German, "Um London Breed den Vorteil des Amtsträgers zu verweigern, wählte der Stadtrat Mark Farrell zum Bürgermeister von San Francisco."Wikilambda will allow contributors to create and maintain functions, their implementations and tests, in a collaborative way. These include the available constructors used to create the abstract content. The functions can be used in a variety of ways: users can call them from the Web, but also from local machines or from an app. By allowing the functions in Wikilambda to be called from wikitext, we also allow to create a global space to maintain global templates and modules, another long-lasting wish by the Wikimedia communities.
In other words, the proposal for what is now called Wikifunctions combined two kinds of functions required for Abstract Wikipedia ("constructors" like the elect example and "translators" or renderers for natural language generation that produce the human-readable Wikipedia text) with a much more general "new form of knowledge assets", functions or algorithms in the sense of computer science. While the examples Vrandečić highlighted in that April 2020 Signpost article are simple calculations such as unit conversions that are already implemented on Wikipedia today using thousands of Lua-based templates (e.g. {{convert}} for the inches to centimeters translation), the working paper published earlier that month (recommended in his Signpost article for "technical aspects") evokes a much more ambitious vision:
Imagine that everyone could easily calculate exponential models of growth, or turn Islamic calendar dates into Gregorian calendar dates. That everyone could pull census data and analyze the cities of their province or state based on the data they are interested in. [...] To analyze images, videos, and number series that need to stay private. To allow everyone to answer complex questions that today would require coding and a development environment and having a dedicated computer to run. To provide access to such functionalities in every programming language, no matter how small or obscure. That is the promise of Wikilambda.
Wikilambda will provide a comprehensive library of functions, to allow everyone to create, maintain, and run functions. This will make it easier for people without a programming background to reliably compute answers to many questions. Wikilambda would also offer a place where scientists and analysts could create models together, and share specifications, standards, or tests for functions. Wikilambda provides a persistent identifier scheme for functions, thus allowing you to refer to the functions from anywhere with a clear semantics. Processes, scientific publications, and standards could refer unambiguously to a specific algorithm.
Also, the creation of new development environments or programming languages or paradigms will become easier, as they could simply refer to Wikilambda for a vast library of functions. [...]
Indeed, a function examples list created on Meta-Wiki in July 2020 already lists much more involved cases than unit conversion functions, e.g. calculating SHA256 hashes, factorizing integers or determining the "dominant color" of an image. It is not clear (to this Wikimedian at least) whether there will be any limits in scope. Will Wikifunctions become a universal code library eclipsing The Art of Computer Programming in scope, with its editors moderating disputes about the best proxmap sort implementation and patrolling recent changes for attempts to covertly insert code vulnerabilities?
This ambitious vision of Wikifunctions as spearheading a democraticizing revolution in computer programming (rather than just providing the technical foundation of Abstract Wikipedia) appears to fuel a lot of the concerns raised in the Fellows' evaluation, and conversely motivate a lot of the pushback in the Abstract Wikipedia team's answer.
Would adapting existing NLG efforts mean perpetuating the dominance of "an imperialist English-focused Western-thinking industry"?
Another particularly contentious aspect is the Fellows' recommendation to rely on existing natural language generation tools, rather than building them from the ground up in Wikifunctions. They write:
Clearly, as Denny pointed out, the bulk work of creating NLG renderers would fall on the community of Wikipedia volunteers. [...] Hence, the necessity of a collaborative development and computation environment such as Wikifunctions.
While the core argument is correct, the fallacy lies in the scope of the Wikifunctions project, which is intended to cover any conceivable computable function (and also using various implementation languages [...]). Since NLG renderers are specific types of functions (transforming specific data types into text, possibly using specific intermediate linguistic representations) it would suffice to create a platform which allows creating such functions. There are many extant NLG systems, and some, such as Grammatical Framework, already have a vibrant community of contributors. Instead of creating a novel, general computation system such as Wikifunctions, it would suffice to create a collaborative platform which extends one of these existing approaches (or possibly creating a new NLG system adapted to the scope and contributor-profile of Abstract Wikipedia, as suggested by Ariel Gutman).
The Foundation's response argues that this approach would fail to cover the breadth of languages envisaged for Abstract Wikipedia:
Some of our own colleagues, like Maria Keet, have noted the inadequacy of those systems for many of the languages Abstract Wikipedia is intended to serve.
In particular, according to Keet, Grammatical Framework notably lacks the flexibility to handle certain aspects of Niger-Congo B languages’ morphology. The traditional answer to this kind of critique would be to say, “Let’s just organize an effort to work on Grammatical Framework.” The design philosophy behind Abstract Wikipedia is to make as few assumptions about a contributor’s facility with English and programming experience as possible. Organized efforts around Grammatical Framework (or otherwise) are not a bad idea at all. They may well solve some problems for some languages. However, the contributor pools for under-resourced languages are already small. Demanding expertise with specific natural languages like English and also specific programming paradigms contracts those pools still further.
(However, in a response to the response, Keet – a computer science professor at the University of Cape Town who volunteers for Abstract Wikipedia – disputed the Foundation's characterization of her concerns, stating that her "arguments got conflated into a, in shorthand, 'all against GF' that your reply suggests, but that is not the case.")
The Abstract Wikipedia team goes on to decry Grammatical Framework as a –
[...] solution designed by a small group of Westerners [that] is likely to produce a system that replicates the trends of an imperialist English-focused Western-thinking industry. Existing tools tell a one-voice story; they are built by the same people (in a socio-cultural sense) and produce the same outcome, which is crafted to the needs of their creators (or by the limits of their understanding). Tools are then used to build tools, which will be again used to build more tools; step by step, every level of architectural decision-making limits more and more the space that can be benefitted by these efforts.
Grammatical Framework would probably give a quick start to creating the basis for a NLG system perfectly suited to "about 45 languages of various linguistic families" – less than 10% of all existing written languages. However, while designing a system that is explicitly focused at covering the knowledge gap of languages that are underrepresented on the Internet, basing the architecture on a framework that gives support to less than the 10% most represented languages defies – by design and from the very start – the bigger purpose of the project.
Discuss this story
Speaking of the Board, one of the current trustees who appears to be most informed about the project may be the recently re-elected Shani, she has been conducting several interview-like conversations with Denny about it. This part (from May 2022) is interesting with regard to the question about the scope of Wikifunctions: In response to a question about what Wikifunctions is and why it is needed, Denny spends several minutes explaining the pedestrian Wikipedia-focused use cases like calculating a person's age, but never mentions the expansive vision of democratizing programming, providing a platform "where scientists and analysts could create models together", etc. Regards, HaeB (talk) 19:12, 1 January 2023 (UTC)[reply]
I'm always disappointed when a response is entirely defensive. It offers a terrible start at a dialog, if nothing else, and will predictably divide responses into polarized sides. I sympathize more with the critique than the response, myself, and so perhaps this isn't a neutral reaction, but I really wish the official foundation response had found opportunities to embrace criticism and a few opportunities to admit that a change in direction might be warranted. Ie, a "these are the critiques we feel are more valid" instead of a blanket "none of the critiques are valid". You don't have to agree, but offer a counterproposal at least. Doubling down on the original plan with no changes after the initial years experience seems to indicate management failure, regardless of the technical merits. No project survives initial implementation completely unchanged. C. Scott Ananian (talk) 15:43, 1 January 2023 (UTC)[reply]
I feel like the team's response to the criticism kind of missed the mark. The criticism raised some risks, and then suggested some solutions. The response seemed to focus on the suggested solutions and why they didn't go with them originally, which isn't what i would call the meat of the criticism. The meat of the criticism comes down to pretty generic concerns - too much waterfall, not agile enough (e.g Trying to do everything all at once), too much NIH syndrome, too much scope creep, not enough focus on an MVP. These are all very common project management risks to focus on in the tech world, and there are many solutions. The critics suggest one possible thing to do, but certainly not the only things possible. I would expect a response to something like this talk about how those risks will be mitigated (or dispute the significance of these risks), not just talk about how they don't like one potential solution. I also am pretty unconvinced with the appeal to avoid cultural bias. Not because i dont think that is important, but because it is being treated as a binary instead of something to minimize. Yes, its an important thing to try to reduce, but you will never fully eliminate as everything anyone done is informed by cultural context. It is a risk that needs to be balanced against other risks. You can't think of it as something to eliminate entirely as the only way to do that is to do nothing at all. Bawolff (talk) 22:05, 1 January 2023 (UTC)[reply]