Wikipedia:Autoconfirmed article creation trial/Post-trial Research Report

Report written by: Morten Warncke-Wang, with Ryan Kaldari and Danny Horn

The Autoconfirmed article creation trial (also known as "ACTRIAL") is a six-month trial that ran on the English Wikipedia from September 14, 2017 to March 14, 2018. During the trial, article creation was limited to users with autoconfirmed status (at least ten edits and at least four days since registration). Studying the first two months of the trial, our findings are:

• No apparent effect on new user activity levels and retention.
• A shift in content creation from the article namespace to the Draft namespace, with a subsequent shift in review workload from New Pages Patrol to Articles for Creation. This shift leads to the latter reviewing process struggling to keep up with an increasing backlog of review requests.
• There is a reduction in unencyclopedic content being created in the article namespace.

We expand on these findings and provide key questions for follow-up discussions both amongst members of the English Wikipedia community as well as the Wikimedia Foundation.

For a more detailed analysis and breakdown, please see Research:Autoconfirmed article creation trial.

Introduction

Screenshot of the New User Landing Page.

The Autoconfirmed article creation trial (also known as "ACTRIAL") is a six-month trial that ran on the English Wikipedia from September 14, 2017 to March 14, 2018. During the trial, article creation was limited to users with autoconfirmed status, meaning they had made at least ten edits and the account was at least four days old. When new editors tried to create a new page, they were redirected to a landing page offering links to the user's sandbox, a task recommendation page, and using the Article Wizard to create the page in the Draft namespace. Prior to the trial, article creation was available to anyone with a registered account.

This trial has a long history. Restrictions on who could create articles on the English Wikipedia were first put in place in December 2005, when Jimmy Wales announced (see also Signpost coverage) that users without accounts could no longer create articles. A proposal to further restrict article creation to users with autoconfirmed status was put forth and passed in 2011. A second proposal determined that the trial should run for six months. The Wikimedia Foundation declined to implement this (T32208), and instead helped develop the Page Curation extension to make it easier to patrol new article creations.

The trial resurfaced again in 2017, at which time an ad-hoc consortium of employees of the Wikimedia Foundation did a study of the backlog of articles needing review and published a report. After discussing this report with the community, the WMF agreed to implement the trial in collaboration with the community and study the results. The WMF devoted resources and hired a researcher to work on designing the study and perform the analysis of the effects of the trial. The study was designed in collaboration with the community on both the English Wikipedia and Meta-wiki in order to determine how to best study the effects of the trial.

The trial started on September 14, 2017 (at approximately 22:30 UTC) and ran until March 14, 2018. This report is based on an analysis of the first two months of the trial, comparing statistics for that period against historical data. We will also make observations about later developments during the trial where appropriate.

In summary, our key findings are:

• The trial has no apparent impact on the activity levels and retention of new users.
• Content creation has shifted from the article namespace to the Draft namespace, with a subsequent shift in reviewing from New Page Patrol to Articles for Creation. Where New Page Patrol previously reported issues with their backlog, Articles for Creation appears to struggle with similar issues during the trial.
• Changes in content quality during the trial are mainly found in a reduction in creation (and subsequent deletion) of unencyclopedic content in the article namespace. There is also a small increase in deletions of similar types of content from the Draft namespace.

Our study is organized into three themes corresponding to the main findings above: new user activity and retention, Wikipedia's quality control processes, and content quality. We go into our findings for these three areas in more detail below, before providing some suggestions for the English Wikipedia community and the Wikimedia Foundation aiming to spur a fruitful discussion of how the community should proceed and how the WMF can help improve content creation processes on Wikipedia. Readers who seek more detailed results and analysis can find more information on our research page on Meta-wiki.

New user activity and retention is largely unaffected

Our first set of hypotheses focuses on how newly registered users are affected during ACTRIAL. There are a variety of ways to measure activity on Wikipedia, and these hypotheses cover most angles. Two key measurements of concern are user retention (surviving editors, Hypothesis 5) and overall user activity level (average number of edits, Hypothesis 7). We would be particularly concerned if newly registered accounts were less likely to stick around during ACTRIAL, or if they tend to make fewer contributions to the encyclopedia.

Graph of number of accounts registered monthly. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.

We find that ACTRIAL has no measurable effect on new user registrations and activity levels. During the trial, the number of monthly registrations holds steady (Hypothesis 1). The average number of registrations for Sept–Nov 2017 is 151.0k, or about 5,000 accounts per day. This is a moderate reduction of 1.7% from 2016's average of 153.6k for Sept–Nov, and well within forecasted expectations.

Secondly, we also find that the trial has no effect on activity levels of new accounts in the first 30 days after registration. There is no difference in the proportion of accounts making at least one edit (Hypothesis 2 ), the average number of edits made (Hypothesis 7), the proportion of accounts reaching autoconfirmed status (Hypothesis 3), median time to autoconfirmed status (for accounts reaching it, Hypothesis 4), and the diversity of work done (average number of namespaces or pages edited, Hypothesis 6).

Graph of the monthly proportion of surviving new editors. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.

Where we do see changes during the first two months of ACTRIAL is new user retention, measured by the rate of surviving new editors (accounts who make an edit in their first and fifth week since registration). Looking at all new accounts, we find an increase in retention to a daily average of 3.3% (compared to 2.8% across similar periods of the years 2014–2016; Hypothesis 5). One thing to keep in mind is that during this period, September to November, we typically see an increase in retention, likely due to the school cycle. It is therefore unclear whether ACTRIAL is the cause of this increase.

Since there is a restriction on article creation in place during the trial, we further segment accounts into those that created an article/draft and those that did not. For accounts that did not create an article or draft, we find an increase in retention similarly as described above. When it comes to accounts creating an article or draft, we focus on those creating drafts. We do this partly because we have found that those accounts would make those creations usually within the first 24 hours after registration, which is impossible during ACTRIAL. Secondly, the trial puts in place an increased threshold, both time- and activity-based, for those who want to create articles, which previous research indicates is associated with increased retention. This makes a comparison of retention for article creators less interesting.

For accounts that create drafts we find a decrease in retention compared to 2015–2016 from 5.2% to 4.6%. However, as we will discuss below, there is a shift of content creation from articles to drafts during ACTRIAL, meaning that a direct comparison is problematic. If we instead hypothesize that draft creators during ACTRIAL are a group consisting of those who would have created drafts and those who otherwise would have created an article, then we find no significant change.

In summary, our results suggest that ACTRIAL has little to no effect on new user participation. We do find an increase in new user retention among users that do not create an article or draft, but are uncertain whether that is caused by ACTRIAL or just random variation.

Shift in content creation and review

Our second set of hypotheses focuses on the English Wikipedia’s quality assurance processes, in particular two processes that review content: New Pages Patrol (NPP), which reviews articles that are created, and Articles for Creation (AfC), which reviews drafts. We find a shift in content creation from articles to drafts, meaning we see a shift in workload from NPP to AfC. Following that shift, we see indications that AfC struggles to keep up with the increased number of submissions, while NPP gets some breathing room and substantially reduces their backlog.

Following the start of ACTRIAL, the rate of article creation is reduced in concordance with the removal of non-autoconfirmed article creations. This reduction does not affect overall article growth, where we find no change (Hypothesis 15). An analysis by WMF staff prior to ACTRIAL starting indicated that a large proportion of non-autoconfirmed article creations were deleted, and during the trial these are no longer created. We also investigated the deletion rate of article creations by autoconfirmed users, and again find no change (Hypothesis 14).

Backlogs of work needing to be done is a consistent challenge in Wikipedia. One of them is the backlog of articles needing review, which when ACTRIAL started contained more than 14,000 articles; the issue of keeping up with the influx of articles being created was arguably one of the motivations for the trial. During the trial, this backlog has been reduced considerably, mainly accomplished during a backlog drive from mid-December through January. Prior to and after that drive, there were long periods of general stability in the backlog. Typically, patrolling work is in balance with the influx of articles being reviewed, something we also find is the case for the first part of the trial (related measure of Hypothesis 9).

For the Draft namespace, we see a significant increase in page creations together with an increase in submissions to AfC (Hypothesis 16). We see indications that people reviewing Articles for Creation struggle with keeping up; their backlog of pending reviews increases more rapidly during the first two months of ACTRIAL than any previous period in our dataset (Hypothesis 17). They continue to be “severely backlogged”, per the description of their backlog on the AfC project page.

This shift of content creation from the article to the Draft namespace is worrisome. Previous research has shown that the AfC process is not as collaborative as creating content in the article namespace. It’s problematic if Wikipedia derails contributions by new users into a space where collaboration does not happen, leaving the newcomer to figure everything out on their own.

• 
  Graph of the number of articles in the New Pages Patrol backlog during the trial. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.
• 
  Graph created by Insertcleverphrasehere showing the age of articles in the New Page Patrol backlog as of March 12, 2018. Green signifies articles younger than one month, orange articles are one to three months old.
• 
  Screenshot of the AfC project page on March 13, 2018, showing the "severely backlogged" status.
• 
  Graph of AfC submissions per day from the Draft namespace. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.

Less low-quality content in article space

Multiple indicators point to a significantly lower influx of unencyclopedic content in the article (Main) namespace. Some of this content appears to be created in the Draft namespace. For articles passing our "encyclopedic content" criteria, there is not a significant change in quality. We also find that articles do not gain quality more rapidly during their first 30 days since creation. In other words, ACTRIAL appears to be successful in reducing the creation of low-quality articles that would typically have been deleted.

We used several indicators of content quality. First, we measured the proportion of pages that are permanently deleted. If a page is permanently deleted, it obviously contained content that is unfit for the encyclopedia (e.g. copyright infringement). Secondly, we measured the proportion of pages that are labelled as "OK" by ORES' draft quality model. The draft quality model is trained on linguistic features to identify spam, vandalism, and attack pages, and we chose an appropriate threshold in order to maximize prediction performance. Lastly, for articles that were labelled "OK", we measure their quality using ORES' WP 1.0 quality model. That model predicts which of the English Wikipedia's quality assessment classes an article belongs to.

Proportion of permanently deleted articles created from January 1, 2016 onwards.

Using these methods to measure article quality, we find a significant reduction for the indicators associated with unencyclopedic content: permanently deleted articles, and articles not flagged as "OK" by the draft quality model (Hypothesis 20). This finding is echoed in our analysis of reasons for why articles get deleted (Hypothesis 18). During ACTRIAL, we see a significant decrease in the average number of deletions per day, and this reduction mainly comes through speedy deletion criteria.

We also made a similar analysis of pages created in the Draft namespace (Hypothesis 19). Given the shift in creations to that namespace, it is important to know whether the content created there is different from what we previously saw in the article namespace. Here we find a small but significant increase in permanent deletions, an increase in the rate of deletions, and that some of this increase comes through deletion of unencyclopedic content (e.g. advertisements). This increase in deletions is not commensurate with the increase in draft creations, meaning that we see a lot of created drafts that appear to not warrant deletion.

In summary, we see a significant reduction in deletion of unencyclopedic content during ACTRIAL, but otherwise no change in content quality. Some of the unencyclopedic content appears to have moved to the draft namespace. To what extent ACTRIAL discourages disruptive contributions is unknown and would require further study.

Suggestions to the English Wikipedia Community

A key question for the community following the trial is: what should Wikipedia’s publishing model be? The Wiki Way is to publish instantly, but make it easy to undo. The restrictions on article creation made by ACTRIAL shifts the model to review-then-publish for many accounts. Research on AfC has found that going through that process means drastically less collaboration than creating an article directly in the main namespace. Is that beneficial to Wikipedia? If the community decides that article creation should be restricted, is autoconfirmed status a good threshold? One example where that restriction hinders contributions is if an experienced contributor comes in from another Wikipedia. Where they previously could create an article (e.g. a translation of one of theirs), it would now have to go through AfC or be created as a draft in the user namespace and later moved, both reducing the opportunity for collaboration and improvement.

How can the community encourage and reward maintenance work such as reviews at New Page Patrol and Articles for Creation? There is no doubt that work on encyclopedic content is important, but maintenance work is also important. They are a vital part of Wikipedia’s quality assurance processes. The February 20, 2018 Signpost news and notes mentions that there were no Requests for Adminship in January 2018. Being an admin is performing maintenance work. Studying ACTRIAL, we see some of the challenges with sustaining these types of communities, e.g. being able to keep up with the influx of articles or drafts needing review. It seems quite clear that switching off article creation for a group of potential contributors does not solve this problem, meaning that the community should carefully consider how it values and rewards maintenance work, and how it can maintain a healthy community of maintainers.

Suggestions to the Wikimedia Foundation

Our suggestions to the Wikimedia Foundation all relate to a key question: how can Wikipedia’s article creation process be improved? We discuss three perspectives on the process, and these perspectives seek to determine where the pain points in article creation are, and how they affect whether we retain promising new contributors.

We have found a shift in content creation during ACTRIAL from the article namespace to the Draft namespace, and submissions of drafts for review at AfC. Prior to our analysis, little was known about AfC. While we now know more about AfC, there are many open questions. How good is the AfC process? Who uses it, and who reviews the drafts? What are the key social and technological parts of it, and can these be improved? The focus during ACTRIAL has perhaps been on NPP, a process which the WMF previously devoted some development resources. With our findings that AfC struggles to keep up with the influx of review requests, it is time to devote some resources there too.

Creating an article that does not get deleted is a difficult task. Can the WMF help design the process in such a way that someone who wants to create a legitimate article is more likely to succeed? While not the primary focus of our analysis, during our work we discovered that the publication rate of pages created in the Draft namespace is incredibly low (about 1.2%), and that a substantial proportion of articles created by fairly old accounts get deleted. These both suggest that article creation is challenging. How can that process be improved? Are there social aspects of it that become blockers, and how does technology fit into the picture?

A key question that our study does not answer is: how are we losing promising contributors? At what point in their time on the site do promising contributors decide to leave, and why do they do so? Prior to ACTRIAL, we saw a lot of articles created by newly registered accounts. Not all of those articles got deleted. Who were the creators of those? Did these creators show up to Wikipedia during ACTRIAL but left when they were unable to create the article? Maybe these new contributors were creating articles that otherwise would not be created, e.g. articles in areas that are underrepresented on Wikipedia. With the 2017 movement strategy’s focus on “knowledge equity”, knowing if content goes missing when article creation is further restricted is of the essence.

There is a great opportunity for the WMF to make substantial contributions to both the research literature on online communities as well as the Wikipedia community by devoting resources to studying the article creation process and how new contributors are welcomed to Wikipedia.