Jump to content

Wikipedia talk:Autoconfirmed article creation trial/Archive 3

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Archive 1Archive 2Archive 3Archive 4Archive 5Archive 6

Landing page proposal

We took some of the ideas from MusikAnimal's newbie editor notice (and the edit-a-thon discussion above) and created a rough draft of what we might put on the landing page for non-autoconfirmed users that try to create an article. You can see the mock up here: Wikipedia:Autoconfirmed article creation trial/Landing page mock-up. The final landing page will actually be a special page that will collect stats on which options the user chooses. Please give us feedback on this mock-up and let us know what should be changed, added, or removed. One of the goals for the mock-up was to keep it limited to no more than 5 different links and no more than a few sentences of text. That way users might actually read it and we will avoid duplicating content that is already in the Article Wizard or Wikipedia:Your first article. Let us know what you think. Kaldari (talk) 00:36, 21 July 2017 (UTC)

Kaldari, two questions: first I meant to ask above, are we still creating a new user permission to create pages that only autoconfirmed have or are we just redirecting to the landing page? I thought the consensus previously was for "option 3", but if we confirmed this I missed it. Second, this might sound petty, but do we have any better images than the puzzle-head guy? It's a bit dated (gives me pleasant memories of the Microsoft office paper clip). I'm the absolute worst because I'm not an illustrator of graphic designer so I can't make anything myself, but I think having a more current image would be less intimidating to newer users by implying the software was up to date. Thanks to you and the Community Tech team for all your work. TonyBallioni (talk) 00:45, 21 July 2017 (UTC)
I think the first question is superfluous, TonyBallioni. The whole idea of ACTRIAL is first and foremost that it prevent non-confirmed users from creating pages (I hesitate to call them 'articles' then most of them are not) in mainspace. If the WMF doesn't do it , I will - then there will follow the mutiny I warned about and the WMF will finally lose face for ever, and with it the reputation of Jimbo Wales' Wikipedia, and to heck with my own reputation. Nobody pays me for what I do here so I haven got a job to lose - just a rather warped hobby. Kudpung กุดผึ้ง (talk) 03:12, 21 July 2017 (UTC)
The landing page is nothing new. It's a direct copy of the one that already existed, except that the nice rounded-corner buttons have been changed for the ugly rectangular ones (that's another effect of the MediaWiki people changing the CSS without consulting the readers and editors). The situation with the arrival of rubbish pages is a lot worse now than it was when we first wanted ACTRIAL. What is need here (in order to comply with modern Communication Studies, is a message directly n the registration page like this which implies a firm instruction but from a psychologically positive approach:
If you are going to create a new article, we have put in place some exiting features to help you through the process and it will be seen by visitors to Wikipedia as soon as it meets our minimum standards for display.
Kudpung กุดผึ้ง (talk) 03:35, 21 July 2017 (UTC)
That's right. Two distinct issues have often been conflated in these discussions of "landing page": what (if anything) we tell new editors when they go to create their first article, and what (if anything) we tell them at the earlier stage when they are registering their account (let's call this the "registration page" or "sign-up screen" instead). About three-quarters of new accounts that edit at all start by editing an existing article, and it must be discouraging not only to have your new article deleted but to have your edits reverted, in both cases just because you'd never been told that there were any restrictions. Not to mention all the work for the rest of us in spotting and dealing with these edits from newbies, and if we're nice going to their talk page and so on.
However, I'm not sure that the issue of the sign-up screen belongs under ACTRIAL at all. Because what is on this screen, I think, shouldn't cover just new article creation, but editing in general. The contents of the sign-up screen are apparently under our control in en-wiki (see User:Noyster/Sign-up_screen). JohnCD, unfortunately no longer with us, used to argue trenchantly for restoring some guidance there.
The snag is that any tweaks there now could confound the results of ACTRIAL. Perhaps we can return to this issue straight after ACTRIAL is done and evaluated. (Spell it exciting though, we don't want to suggest "exiting" to them, at least not yet): Noyster (talk), 10:32, 21 July 2017 (UTC)
I think it is also important that, while we generally assume good faith, a sizeable number of non-autoconfirmed page creators have some form of conflict of interest. Although I can't give you COI stats (as I doubt those exist), through my own experience in reviewing pages new pages these statements hold true. This is relevant to the landing page design because such non-autoconfirmed users with conflicts of interest won't be interested in editing Wikipedia beyond getting their spam into the encyclopedia. So my view would be to try and push them through the "reading the policies" process to ensure that by the time they come to starting a draft or gaining autoconfirmed status, they have an understanding of WP:NOT. DrStrauss talk 11:37, 21 July 2017 (UTC)
DrStrauss: in the landing page mock-up, the Article wizard is one of the three main options. The Article wizard includes information about COI, notability, sources, NPOV and several other policies/guidelines. I think it makes sense to put that information in front of people who are specifically trying to create a new page. I'm not sure about giving it to everyone as soon as they register; new people registering might just want to make a helpful edit. -- DannyH (WMF) (talk) 20:31, 21 July 2017 (UTC)
I said this at the discussion about MusikAnimal's original design and I still believe that a link to IRC should be on the notice. jcc (tea and biscuits) 20:34, 21 July 2017 (UTC)
@Jcc: Thanks for the suggestion. I've added a link to IRC to the mock-up. Kaldari (talk) 22:55, 21 July 2017 (UTC)
@TonyBallioni: The current idea would be to automatically redirect all users without the autoconfirmed right to this special landing page and force them to create articles either in Draft space (via the Article Wizard) or in their user sandbox. We could create a new `createcontentpage` right, but that would require some significant refactoring of MediaWiki code (and delay ACTRIAL). I'm not sure what you mean by "option 3". Do you mean option 2 from above about creating an event coordinators user group that could grant the autoconfirmed right to new users? I think that suggestion got a lot of support and I think we should move ahead with an RfC on it. The landing page proposal would still be compatible with that, and I don't think we would need to delay ACTRIAL for the RfC. If no one objects to the idea of an RfC for a new event coordinator user group, I'll probably go ahead and create the RfC next week. Kaldari (talk) 23:09, 21 July 2017 (UTC)
Kaldari, sorry for the late reply. I was referring to the possible ways to implement that you proposed at WT:NPPAFC. Take the existing work that was done on Extension:ArticleCreationWorkflow, reduce the scope, and deploy a simplified version of it. Reading through it again, I seem to have misread it as saying the method of preventing page creation would also be as proposed in 2011. I suppose my main question is would there be any backdoor to skip the redirect? TonyBallioni (talk) 21:32, 22 July 2017 (UTC)
@TonyBallioni: Yes, currently we are moving ahead with option 3 (implementing a simplified version of Extension:ArticleCreationWorkflow), but it will function based on the existing autoconfirmed user right. If ACTRIAL ends up becoming permanent, however, we will probably want to create a new user right specifically for content page creation (since otherwise things like IP rate limiting will be forever coupled with content page creation, which doesn't really make sense). Per the consensus above, there will be no backdoor to work around the redirect. Kaldari (talk) 17:51, 23 July 2017 (UTC)
Kaldari, got it. That all makes sense now. The misunderstanding comes from my knowing what the end objective is and not the way of explaining the technical way to achieve it with precision. Thank you for clarifying. TonyBallioni (talk) 23:12, 23 July 2017 (UTC)

Graphics

@Kaldar and DannyH (WMF):, I'm sure you're tired of me as a pest on this by now, but does the WMF have a graphics team that could update the flow/images of the landing page to be more current? Or are there graphics on Commons that we could use. I really do think that giving that page a refresh in terms of visual appeal will make it easier for good faith new users to become involved in Wikipedia. TonyBallioni (talk) 20:50, 26 July 2017 (UTC)

Hi TonyBallioni, that's a good thought. We don't have a designer on the Community Tech team, but I'll ask one of the designers on another team if he's got time to take a look. I'll let you know what he says... -- DannyH (WMF) (talk) 21:34, 26 July 2017 (UTC)
Yes, I know how comms departments are often stretched, but I do think the first page many new users will see is a type of PR for the Foundation. To be honest, I would love it if the Foundation communications team were involved in the process here to help us communicate a good welcome message to people. TonyBallioni (talk) 02:02, 27 July 2017 (UTC)
  • I have now written this a couple of other places, but this is the place to say it. The landing page should make this process seem completely normal, like drivers ed or like new employee orientation at a company. Very welcoming, very normal -- "Welcome to WP! Working in WP is complicated so we help new editors create their first few articles with a guided process, and then a review, before their articles enter the encyclopedia. Please select a button to begin!" Something like that. Jytdog (talk) 03:13, 19 August 2017 (UTC)
Jytdog, DocJames, Kudpung: The current mock-up for the landing page is here: Wikipedia:Autoconfirmed article creation trial/Landing page mock-up. That's the text that we're planning to use. Let me know if you want to make changes to that text -- we can totally do that, we just need to know what you'd like it to say. :) -- DannyH (WMF) (talk) 19:08, 21 August 2017 (UTC)

Jytdog@TonyBallioni, ironically WP:YFA is the first thing that scares new users away. It does exactly the opposite to its intended effect. We knew 40 years ago from our research at the TU Berlin that there are two possible reactions to such a wall of text (now in a 21st century context) :

  1. Sheesh! if this is what I have to do or know, I won't bother. I'll put my article on WordPress instead.
  2. Bugger this for a lark! I'll write my crap page anyway. Nobody knows who I am.

WP:YFA has been on my 'to do' list for years. It needs completely rewriting, condensing to a third of its current TL;DR, and a very strong emphasis to use the Wizard. I would do it, but what with watching what's going on, and partly rewriting the Wizard pages so that non native users (50% of new accounts) can understand them, there is a limit to what I can do in the time left. I'll help with it though. Kudpung กุดผึ้ง (talk) 22:25, 21 August 2017 (UTC)

  • Responding to Kudpung's ping in this section since it's graphic related: I know Drewmutt had some thoughts on the graphics in the past: he'd probably be able to provide better feedback than I would. I'd just prefer something more contemporary and that doesn't look like Microsoft clipart from 2006 (no offense to Puzzly, but he reminds me a lot of the Office Paperclip). TonyBallioni (talk) 02:45, 22 August 2017 (UTC)

YFA

Kudpung, I looked at WP:YFA to see the edits that you made, but I don't see anything in the history. Did you publish your new version? -- DannyH (WMF) (talk) 16:55, 22 August 2017 (UTC)
No. It's in a user sub page. Kudpung กุดผึ้ง (talk) 18:22, 22 August 2017 (UTC)
Oh, I see. I think the DO and DON'T sections in your version are a little confusing -- in a DON'T list, the items are usually the things people shouldn't do -- DON'T write an article about yourself, DON'T repeat an article that already exists, etc. In your list, it looks like you're saying DON'T consider registering an account, DON'T search Wikipedia first, DON'T practice first. -- DannyH (WMF) (talk) 20:17, 22 August 2017 (UTC)
Does it? Kudpung กุดผึ้ง (talk) 23:56, 22 August 2017 (UTC)
Yes it does, doesn't it? Looks as if my last save failed to load - it happens here. One of the problems is that 9t's suc a WoT it's difficult to know where to begin. Perhaps Jytdog would like to have a go. Kudpung กุดผึ้ง (talk) 00:47, 23 August 2017 (UTC)

Landing page text

@Kaldari and DannyH (WMF): I've made some minor changes. Nothing is undoable. I have no opinion on the graphic. Perhaps Tony has an idea, there are several million images at commons to search through...Kudpung กุดผึ้ง (talk) 22:48, 21 August 2017 (UTC)

Kudpung, you added the sentence "it will be seen by visitors to Wikipedia as soon as it meets our minimum standards for display" to the landing page. Is there a description somewhere of what those minimum standards are? It's a vague phrase, and I think it's potentially confusing, if we don't have anything specific to link to. Also pinging Jytdog and TonyBallioni. -- DannyH (WMF) (talk) 15:14, 22 August 2017 (UTC)
Those would be WP:N, WP:NOT, and WP:BLP. Not sure how you would work those into the text. TonyBallioni (talk) 15:48, 22 August 2017 (UTC)
Don't forget WP:V Legacypac (talk) 16:03, 22 August 2017 (UTC)
Hmm. It's tough figuring out what to include, which is why every effort to do an intro for new people ends up as a "wall of text". :) -- DannyH (WMF) (talk) 16:52, 22 August 2017 (UTC)g
The whole problem of Wikipedia is that it puts people off with rules, regulations, acronyms, and a steep learning curve right in their face at the very beginning. That's why new users don't stay. Vague is best at that stage - after all it's only a splash page. The Wizard will confront them will walls of text soon enough. Kudpung กุดผึ้ง (talk) 18:12, 22 August 2017 (UTC)

Suggestions to improve article wizard

There are a few suggestions to improve article wizard at Wikipedia talk:Article wizard that may be of interest. Thanks. Darylgolden(talk) Ping when replying 06:57, 27 August 2017 (UTC)

Preliminary analysis of patroller workload and article creation

Updated Aug 26, 2017: I've gathered better data on article creations after I found out our previous data did not include deleted articles. This affected the analysis for findings #1 and #2 below, but the conclusion in finding #2 was unchanged. Updated Aug 30 & 31, 2017: I've gathered better data on the number of patrol actions and active patrollers after being notified that we didn't include patrols done outside of the PageTriage extension. All graphs have been updated to reflect the new data sources. This update particularly affected the number of active patrollers as we found many patrollers user alternative tools. I've updated the quick analysis of each graph in order to reflect the current state of the graphs.

Hi everyone! Over the past few days I’ve been doing a preliminary analysis of data on patroller workload and article creation, and found some interesting results that I wanted to share with you.

Quick background: We are interested in understanding the effect of ACTRIAL on Wikipedia’s quality assurance processes, and as part of that have proposed hypotheses of how this will affect New Pages Patrol. Referring to our project page on meta, we want to know more about the following hypotheses:

  • H9: Number of patrol actions will decrease.
  • H10: Number of active patrollers will decrease.
  • H11: The distribution of patrolling activity evens out.

We also have two related measures for H9 and H10, looking at how the number of patrol actions and active patrollers relate to the number of created articles. I’ll summarize the findings we have that I think are interesting. If you’re interested in more details, I describe our methods on the project page on meta and there are notes and larger graphs in my work logs from Aug 22 and Aug 23. Since this is preliminary research the focus is on general trends, and I’ll try to explain what I’m seeing as we go along.

Finding #1: The number of articles created per day is generally stable. The graph above shows the number of articles created from Jan 1, 2011 to July 1, 2017, as well as publications of drafts from User and Draft namespaces. Since the second half of 2014, the number of articles created per day has been fairly stable and averages around 1,100. The use of draft spaces is stable, although since the start of 2017 there’s a lot more publication from the Draft namespace. From what I’ve been able to find out this is due to AfC working to reduce their backlog.

Finding #2: New Pages Patrol does not appear to keep up with demand. The graph above shows the proportion of patrol actions performed to the number of created and published articles from October 1, 2012 (first full month after the PageTriage extension was introduced) to July 1, 2017. This proportion is rarely above 100%, which means that the backlog is generally increasing.

Finding #3: The number of active patrollers used to be increasing. The graph above shows the number of active patrollers per day from October 1, 2012 to July 1, 2017. Back in mid 2013 there looks to be about 100 active patrollers every day, and the number increases fairly steadily until early 2015. Activity picks up again in late 2016, perhaps due to the forthcoming introduction of the New Page Reviewer user right. Once the NPR right is introduced, number of active patrollers per day is stable at around 75.

Finding #4: Most of the patrol work is done by the most active patrollers. The graph above shows the proportion of patrol actions done by the 25% most active patrollers. Generally these make up 70–90% of all patrol actions. We saw earlier that there’s currently about 75 active patrollers per day, meaning that about twenty patrollers are doing most of the work. I think that’s cause for concern about to what extent patrollers work too much and burn out, but also something that might motivate NPPers to recruit more patrollers and encourage inactive patrollers to participate more.

I hope you found this interesting as well, and feel free to ask questions! Cheers, Nettrom (talk) 23:13, 24 August 2017 (UTC)

  • Yes, it was interesting but the results are flawed. Since November last year we introduced a new user group. Only New Reviewers are allowed to patrol pages. These are the only stats that are now relevant.
Reviewers do not suffer from burn out. This is a myth propagated by the Summer of Research programme a few years ago. Unpaid volunteers only do as much as they are prepared to do. This is something that salaried individuals don't understand. What does happen to reviewers however, is that they are so sickened by the flood of raw sewage they are supposed to treat for free and having to argue with nasty 'customers', they just go back to normal editing. Kudpung กุดผึ้ง (talk) 11:13, 25 August 2017 (UTC)
Happy to hear you found this interesting! The introduction of the user right for reviewing should be studied in order to understand whether it changed behaviour, which requires us to compare it against historical data. If it didn't, which appears to largely be the case, then that means that we can pull from a larger set of historical data to understand what happens during the trial.
I don't see a meaningful difference with how you describe reviewer behaviour and what I call "burnout". In both cases we're talking about reviewers not continuing to review. We do not know whether they're halting it temporarily or for good (and I'm not going to speculate about why, we don't have data on that). Most of the review work appears to be done by a small number of reviewers, meaning that if one (or more) of them drop out temporarily it leaves a large amount of work to the others. Increasing the pool of active reviewers, like I suggest, is one way of reducing that issue. Cheers, Nettrom (talk) 16:57, 25 August 2017 (UTC)
Nettrom, we do encourage participation through the newsletter (and the backlog tends to decrease a bit every time we send one out.) There is a concern here, however, in that part of what both the WMF and the community want is review of new content that does not also bite inexperienced users: we want people to like us. I'm hardly a radical inclusionist, but I've had to "counsel" people on inappropriate CSD and BLPPROD nominations, and my personal "favourite" to deal with is tagbombing with AWB. Finding the right balance between experienced users and more users is difficult.
On a different note, while I appreciate your comments here, I think making suggestions as to how we can deal with the data or what is a cause for a concern and suggestions for how to deal with it gets close to the lead ballon that was floated in May. Part of the reaction to that was it came off as the WMF telling the community how to deal with a complex issue in a way that was seen as counter to the wishes of the community. I think everyone here would agree we need more reviewers, but we also don't think that is the only problem. The community will reach its own conclusions from the data as what is the best way to respond to it and have an RfC on the issue 7 months from now. My concerns here are that we want an RfC that is framed by the community, not by the Foundation, and commentary beyond basic explanation of the numbers in prose might have a way to sway that. TonyBallioni (talk) 17:42, 25 August 2017 (UTC)
TonyBallioni: I can understand why there's a concern about WMF having an "official" interpretation of the data -- there's a history of Foundation people pushing our own interpretations and bias without giving evidence, and without listening to other people's experience and ideas. I think in this instance, Morten is showing the data and where it comes from, and his interpretations and suggestions are part of an ongoing conversation that we're all having about it. Thinking and talking about the data is part of a researcher's job; it's what they do. -- DannyH (WMF) (talk) 20:28, 25 August 2017 (UTC)
Yes: I think there is healthy room for discussion here, but I also did want to point out that this is sensitivity on the part of the community on this. Thanks for the response, Danny. TonyBallioni (talk) 20:49, 25 August 2017 (UTC)
I'm just seriously concerned about avoiding a repeat of an earlier issue where the WMF promised help with stats on a community NPP project then withheld the results until they had published their own interpretation of it.
I don't wish to speculate whether or not more Reviewers are needed. Again, the WMF has no knowledge of what goes on at PERM - and won't accept empirical findings anyway. Kudpung กุดผึ้ง (talk) 10:05, 26 August 2017 (UTC)

Adding a note here that I've updated the graphs for Findings #1 and #2 after I found out that our dataset did not include deleted articles, the new graphs include deleted articles. The general trend in #1 is somewhat different, the conclusion in #2 is the same. I've added a note at the top as well. Cheers, Nettrom (talk) 23:14, 26 August 2017 (UTC)

2017 Uptick in pages from draft and userspace also reflects a push to clean up stale non-afc draft and user space that continues to surfaced many pages Legacypac (talk) 23:20, 26 August 2017 (UTC)

Legacypac: I read some of the AfC talk page archives and got the impression that some sort of cleanup push was happening. Thanks for confirming that that's the case, appreciate the insight! Cheers, Nettrom (talk) 14:54, 27 August 2017 (UTC)
Yes AfC has been pushing at the backlog that's mostly a regular thing. I'm not convinced that generated the uptick in pages from Draft & Userspace. I'm referencing the net 8-10,000 page reduction here since Nov?? 2015 [1] and the net 6,000 page reduction in non-afc drafts here [2]. Both these backlogs have seen large net reductions, even though new pages are added daily to the backlog. With the expansion of G13 to All Draftspace several days ago, we are quickly driving the MusikBot report to near zero. IE delete, fix and promote, redirect, or send to AfC and hope for improvement. Legacypac (talk) 17:05, 27 August 2017 (UTC)

Comments on the graphs

Hi Nettrom, I've now had an opportunity to examine these graphs more closely. I think they would have bee easier to read if they had been presented as line graphs (we are not all statisticians) and with sample period that starts at the point that we, the volunteer community, are interested in such as the the dramatic sharp rise in early 2016 from 6,000 to 22,000, and the sudden lack of interest by patrollers as manifested in Feb 2017 - for which no theories have been forthcoming.

  • Finding #1: The number of articles created per day is generally stable: How do we explain the marked increase in drafts in Q1 2017 which suddenly dropped off again? Was one draft reviewer doing a lot of reviews? Legacypac has provide some possible explanations but as AfC and NPR are inextricably closely rlated and due to become even more so, we need some factual data about the AfC project and its operations.
  • Finding #2: New Pages Patrol does not appear to keep up with demand. The 2016/2017 figures seem to correlate to the graphs I produced. We still have no explanations for the sudden swings in the mood of the patrollers - particularly in Feb 2017. It would appear that since the beginning of Q2 the percentage is actually on the rise.
  • Finding #3: The number of active patrollers is slowly increasing. I see an improvement beginning around the start of Q4 2016, but the general trend from mid 2017 seems to be towards a slight decline.
  • Finding #4: Most of the patrol work is done by the most active patrollers. This seems to more or less reflect what we have already empirically established. It does not however provide any substance to claims of burn out. I contend that many of the applicants for the NPR right have been hat collectors. Certainly from being one of the most active admins to accord these rights, I find that many applicant are not new users but never did much active patrolling pre Nov 2016 and have no intention of regularly patrolling, they are therefore accorded the right on numerical criteria and a trouble-free history. The regular reviewers are not suffering from burn out because they are still doing it. This is why these graphs would be more helpful if they show a shorter sample period for our current purposes and preferably displayed as a line starting in June 2017 2016 which would then show more clearly what happened when NPR was introduced in Nov 2016 and the trend since.

I perfectly understand however that the WMF, as non-patrollers, wants these stats for their purposes which are also important, while the requirements of the community are quite different. We need to see at the end of ACTRIAL how we can directly address and motivate the idea of New Page Patrolling , and with user feedback, as we already did at Wikipedia:Page Curation/Suggested improvements (which coming up for a year ago has still received little or no formal recognition from the WMF), to know what we want the WMF technicians to develop for us.

The backlog, probably as a result of TonyBallioni's news letters, has dropped from 22,000 to 16,000 where it has again plateaued out for several weeks, but this is very far off being able to be claimed as a success. Kudpung กุดผึ้ง (talk) 02:10, 28 August 2017 (UTC)

  • I'd make a very simple observation re: percentage of work done. This is a basic Pareto principle and would be expected in about anything. My recent review activity has dropped off because I've been focusing more of my time on ACTRIAL and other project-space efforts that would overall improve our systems for handling new content. I'm sure I'm no longer in the 20% because of this, but I'm very much active in the NPP project and wouldn't consider myself burned out. I think a lot of our reviewers such as Rentier also focus their efforts at WP:SPI and WP:COIN, which overlaps at NPP, but is often more time consuming.
    Btw, DannyH (WMF), if you haven't already seen Rentier's NPP browser, it really is a great tool (I always try to brag on our NPP contributors when I get the chance :) ) TonyBallioni (talk) 02:25, 28 August 2017 (UTC)
TonyBallioni, I hadn't seen that yet, thanks for showing it to me! Do you know how many reviewers use it? -- DannyH (WMF) (talk) 17:30, 28 August 2017 (UTC)
I, for one absolutely LOVE it. The ability search for phrases that are common in spam is very useful, as is the fact that it allows me to focus on areas where I have relevant domain knowledge. Mduvekot (talk) 17:50, 28 August 2017 (UTC)
Kudpung Thanks for taking a closer look! I agree that the graphs can be difficult to interpret and would benefit from for example having a trend line added. So far I have not focused on that part of the analysis as I'm instead working on gather more of the data we need. You're welcome to grab the datasets behind this and process them yourself, they're in our GitHub repository, the files you'd want to grab are: enwiki_article_creations_by_day.tsv (date and number of articles created), enwiki_moves_by_day.tsv (date, number of moves from User, and number of moves from Draft), patrolactions_by_day.tsv (date and number of patrol actions), patroller_distribution.tsv (date, user ID, and number of patrol actions performed, some data processing needed to get the top 25% percentile), and patrollers_by_day.tsv (date and number of active patrollers). I use R for my statistical analysis and have the code for all the graphs and everything also in our GitHub repository, but feel free to use software you're familiar with. Cheers, Nettrom (talk) 20:53, 28 August 2017 (UTC)
Nettrom It's probably not accurate to assume that we humble unpaid article writers are computer programmers and statisticians. That's why why it's difficult for us to collaborate with the people who are being paid to do these things. And the WMF wonders why we are cynical. If it weren't for the fact that we (Scottywong and I) mistakenly thought ACTRIAL needed a tweak to MediaWiki to make it work, it would have been done 6 years ago and we wouldn't be having these discussions. Kudpung กุดผึ้ง (talk) 23:24, 28 August 2017 (UTC)
Kudpung: Yeah, I'm sorry that Morten's not responding right away to the requests for trend lines. He's in the middle of gathering more preliminary data -- we talked today, and agreed that his immediate priority is figuring out how to measure article quality -- so he doesn't have time right now to run trend lines on the graphs he's posted so far. He'll definitely get back to that later, once we've got an idea of where all the measurements are coming from. The offer of looking at the datasets was well-meant, especially because people have posted concerns here that folks won't have access to the data. We'll keep informing everyone of the progress with these stats, as we get ready for the experiment -- there will be lots more in the coming days and weeks. -- DannyH (WMF) (talk) 00:24, 29 August 2017 (UTC)
Thank you for the explanation, I don't want you to think I'm criticising Morten's work, but thanks also for finally telling us that ACTRIAL is not actually your priority or the reason you hired him. Article 'quality' per se is not within the remit of ACTRIAL or New Page Review. Our volunteer work here is aimed at reducing the flow of totally inappropriate page creation, quickly recognising what is apt for the encyclopedia and what is not, how to more easily identify the often subtle signs of what is not, and how to best process all new pages. PR is a triage, it's not a field hospital. I'm pinging TonyBallioni in case he would like to comment and because I'm beginning to think my private volunteer time is being wasted. Kudpung กุดผึ้ง (talk) 00:42, 29 August 2017 (UTC)
Kudpung, I think there must be a misunderstanding. There are three main areas of inquiry on the ACTRIAL research page: the impact on new accounts, the impact on the quality assurance process (NPP), and the impact on content quality as a whole. We want to know if making this change results in fewer bad articles and more good ones. I know that you've seen the three areas, because you commented on all the hypotheses here: meta:Research_talk:Autoconfirmed_article_creation_trial#Comments_from_Kudpung. Figuring out how to measure content quality is difficult; it'll take Morten a couple days to get his head around it. Does that make anything clearer? -- DannyH (WMF) (talk) 00:46, 29 August 2017 (UTC)
DannyH (WMF) I don't think there is a misunderstanding. Without wanting to show off, I think I possibly have the broadest overview NPP, it's impact and procedures, and efforts to get it improved, than anyone who is new to these issues since 2010. The research page is what the WMF wants to do; the community already knows that there is going to be an impact on new accounts. We already know that less than 0.1% of accounts have ever edited. We already know that a significant number of accounts are deliberately created for the sole purpose of making mischief. We do not believe for a moment that ACTRIAL will increase the number of good articles - that would be simple wishful thinking and is not a serious expectation.
But those are the empirical conclusions that a) the Foundation does not wish to entertain, probably because it has no practical experience in these areas, and b) are difficult to explain anyway with pure math. That said, ACTRIAL is designed as a measure to protect Wikipedia's content, integrity, and reputation (which is rapidly deterorating) for quality. 'Quality' per se however, is not within the mandate of New Page Review which is a triage (that was even Jorm's first code name for the development of Curation), and at the end of the day, that's what it all about - and helping the Foundation to finally understand the priorities in producing the software enhancements that are needed to do it efficiently.
As I said above, If it weren't for the fact that we (Scottywong and I) mistakenly thought ACTRIAL needed a tweak to MediaWiki to make it work, it would have been done 6 years ago and we wouldn't be having these discussions. It pains me that I have to keep repeating all this. I'm rather disappointed to realise ...his immediate priority is figuring out how to measure article quality -- so he doesn't have time right now that we have been led to believ that money is being spent on 3rd party research for the purpose of helping us implement ACTRIAL - and then being told there is no time due to other priorities, giving us a half finished data and telling us we have to do the rest ourselves. Kudpung กุดผึ้ง (talk) 01:46, 29 August 2017 (UTC)
Kudpung, I'm sorry that we disappointed you today. I'll ask Morten to pause what he's doing, and run those trend lines for you. -- DannyH (WMF) (talk) 02:47, 29 August 2017 (UTC)

DannyH (WMF)I don't want you to assume that this is a personal request from me. I've received plenty of comments off-Wiki that those graphs are not particularly helpful to us here. All we need are trend lines over a more recent sample period, such as from perhaps the beginning of 2016 and which clearly depict the mood of the reviewers.

In 2009 or 2010 when WereSpielChequers and I first realised that it was becoming difficult to keep the backlog down, some of the intake was over 3,000 a day. The articles in those days however, were certainly easier to patrol - most of them were still about traditional encyclopedic topics and the rest was easily identifiable junk. The curious policy of including every soccer player who had ever kicked a ball had not begun, Internet penetration had not significantly reached non English L1 regions in course of development, and not so many people had discovered how easy it is to exploit our unpaid work to make a fast buck or a quick quid.

By the beginning of 2012 it was already half that number, by 2013 it was a third, and it is now hovering at what appears to be an average of around 900. None of any of this is our doing. The crap we get today has to be seen to be believed, but the paid staff won’t do it or won’t listen. They wouldn’t even take notice until we recently announced our intention to roll out ACTRIAL ourselves, at which point they entered stage left in panic with Wikipedia:New pages patrol/Analysis and proposal.

We really do appreciate your engagement at this stage, but we don't want to lose focus on the essentials, because any remedies that are revealed to be necessary after the trial are going to take months, if not years again, to develop. Kudpung กุดผึ้ง (talk) 03:10, 29 August 2017 (UTC)

Kudpung: You're correct, and it's important for us to share as much information as we can. The graphs that Morten posted aren't "actionable" right now -- this is the beginning of the experiment, not the end.
This is preliminary baseline historical data that we're gathering now, for comparison's sake. I would expect that what we see in this historical data should roughly map with your experience, and I think for the most part that it does. For example, you said about #2, "The 2016/2017 figures seem to correlate to the graphs I produced," and for #4, "This seems to more or less reflect what we have already empirically established." That's good, because it means that we're accurately reflecting what's been happening, which (hopefully) means that we'll be able to accurately describe what happens once the trial starts.
We're erring on the side of showing everything that we have, pretty much as soon as we have it, so some of the things you're seeing are rough. I think the first graph that Morten posted last week was wrong -- he realized after posting it that he hadn't taken deleted pages into account, so he re-ran the data and updated the graph and description with the corrected version. You're going to see some rough things right now, because you're getting information in real time, as we work on the project. -- DannyH (WMF) (talk) 03:41, 29 August 2017 (UTC)
DannyH (WMF): For what it's worth, back when we were initially doing research for ACTRIAL, the statistic that we used to judge article quality was whether or not the article was deleted within a certain amount of time after it was created (minimally, a month or two). If the new article wasn't deleted within a few months, then it was assumed that the article was "good enough" to pass WP's standards. If it had been deleted, then we assumed it was junk. We chose this statistic because this is essentially the primary thing we care about when patrolling new pages: weeding out the obviously deletable articles and doing a bit of triage on the rest. The very clear trend we saw (back in 2011) was that nearly 75% of articles created by non-autoconfirmed editors were eventually deleted, whereas less than 20% of articles created by autoconfirmed editors were eventually deleted. I believe this would be a good statistic to track while ACTRIAL is in effect (and for some time after ACTRIAL ends). If, during ACTRIAL, we see a decrease in the percentage of new articles that are deleted within x months after creation, then we should be able to conclude that ACTRIAL has succeeded in increasing the average quality of newly created articles. See User:Scottywong/Article creation stats for the original data from 2011. ‑Scottywong| prattle _ 04:49, 29 August 2017 (UTC)
Scottywong: Yeah, we're definitely looking at what happens to the newly created articles, and how fast they get patrolled vs deleted. We also want to try to estimate the quality of an article even if it hasn't been reviewed yet -- part of the problem is understanding what's in this 16,000 page backlog. We want to figure out if all the pages that should get deleted are actually getting deleted. -- DannyH (WMF) (talk) 05:08, 29 August 2017 (UTC)
That's a thorny problem. Subjectively judging the quality of an article is arguably a task that requires a human (which is why New Page Patrol is still done by humans and not bots). The ultimate judge of whether or not an article is "good enough" is theoretically whether or not it gets deleted. But that assumes that the reviewing and deletion processes are perfect, which they obviously aren't. I can see that you're concerned that there may be some percentage of new articles that are slipping through the cracks and not getting deleted when they really should have been deleted. A few things to consider:
  1. There will be some number of articles that should be deleted but were "missed" for various reasons. Hopefully this number is not statistically significant.
  2. There may also be articles that were deleted, but not because they were of low quality (i.e. for some housekeeping reason). These articles would offset the errors generated in #1.
  3. To reduce error, you need to choose an appropriately long window of time before judging an article's quality, because the longer a crappy article exists, the more likely it is to be noticed and deleted. For example, if you run ACTRIAL, and then the day after the trial ends you begin looking at the percentage of new articles that were deleted, you will have a large error (because there will be articles that were created only a day or two ago that haven't yet been reviewed and deleted). What I'd suggest is taking a look at the age of the oldest article in the NPP backlog of 16,000 articles, and basing your "window of time" on the age of that article. It looks like the bulk of unreviewed articles were created up to 6 months ago.
This is part of the reason that ACTRIAL was originally designed to be a relatively long-term trial of 6 months. It may take some time after ACTRIAL ends to fully understand its effects. ‑Scottywong| squeal _ 06:15, 29 August 2017 (UTC)
Scottywong, MusikAnimal (WMF), might be able to shed some light on this based on the statistics at WP:MANPP. I believe in past conversations with him he said that he believed that the overwhelming majority of new content got deleted within the first 90 days. TonyBallioni (talk) 06:20, 29 August 2017 (UTC)
Scottywong: Thanks for chiming in here with your perspectives on assessing quality and some historical info, those were very helpful! From what you're writing, it appears that we're approaching this from similar angles. We use article survival as a way to understand quality in some of our hypotheses, and have 30 days as our window of "reasonable amount of time" to know if an article is fit for Wikipedia, as that timespan was also used by Schneider, Gelley, and Halfaker ("Accept, decline, postpone: How newcomer productivity is reduced in English Wikipedia by pre-publication review", OpenSym 2014). It's good to see published research connecting with community concepts like this!
We now also have some machine learning tools to assess article quality that I'll be using to gather data. ORES has models for both overall article quality and testing against some CSD criteria. I'll be looking to gather some data from both of those to see how they relate to NPP and AfC. In addition to assessing content quality at the time it reaches the NPP and AfC queues, we also plan to check in on article quality 30 days after creation (provided the article isn't deleted, of course) to understand how these articles develop and whether that changes during the trial. This should augment the survival-based quality metric and thereby provide us with more insight into created articles and how they develop both before and during the trial. Thanks again for comments, and let me know if you have questions about any of this! Cheers, Nettrom (talk) 18:06, 29 August 2017 (UTC)
Nettrom: That sounds good to me, thanks a lot for the explanation. My only caution would be that using a 30-day window for assessing article survival might be on the low side. If the NPP backlog is longer than 30 days, then it stands to reason that many articles may not get reviewed (and possibly deleted) until after the 30-day window.
Additionally, some of our deletion processes can take a long time. Consider this typical example: a new article is reviewed a week after it is created, and the reviewer decides that it should be deleted, but it is not eligible for speedy deletion. So, the reviewer PRODs the article instead. Five days later, the author of the article removes the PROD. The next day, the reviewer nominates the article for deletion at AFD. The AFD doesn't get a lot of participation and gets relisted three times before it is closed with a consensus to delete. Despite the article getting reviewed relatively soon after being created, it still took 34 days for it to meander through our deletion bureaucracy before it was ultimately deleted.
I'm not really sure how large of an error a 30-day window might create, but my gut tells me that a 60- or 90-day window might result in a more accurate analysis. ‑Scottywong| talk _ 22:52, 30 August 2017 (UTC)

A suggestion

Could we ask the members of WikiProject Statistics if they're interested in helping with the analysis of the data? I know R a bit, but I'm no statistician. I'm glad we're collecting data to establish a baseline though. That will prove to be invaluable later on. Mduvekot (talk) 04:41, 29 August 2017 (UTC)

Mduvekot: Yeah, anybody who's interested; that would be great. If you want a link to pass along, meta:Research:Autoconfirmed article creation trial has all of the research questions, and Morten is keeping that updated with the measurements and preliminary data. We'd love to get more feedback and suggestions about how to measure the impact of the trial. -- DannyH (WMF) (talk) 05:03, 29 August 2017 (UTC)

Graphs from 2016 onwards with trend lines

I've created graphs showing the period from Jan 1, 2016 to July 1, 2017, and added a Loess trend line to them. All of these graphs show daily measurements.

Articles created per day, by type
This plot shows the number of articles created per day, split into three: articles created directly in the main namespace, pages moved into main from the User namespace, and pages moved into main from the Draft namespace.

Total articles created per day
This plot shows the total number of articles created in or moved into the main namespace (the sum of the three measurements in the first graph).

Proportion of patrol actions to created articles
This plot shows the proportion of patrol actions (either logged by the PageTriage extension, or patrols done through other tools) to the total number of created articles. This helps to measure how well the page reviewers are keeping up with the creation of articles.

Number of active patrollers per day
This plot shows the number of patrollers who made at least one patrol action on a given day.

Percentage of patrol actions by the most active patrollers, per day
This plot shows the proportion of all patrol actions performed by the 25% most active patrollers on a given day.

I hope these are helpful, let me know if you have any questions about them. Regards, Nettrom (talk) 15:24, 29 August 2017 (UTC)

Thank you Nettrom. This is much more helpful although the x and y axes could do with some finer graduations. The graphs now all speak for themselves and clearly reflect the curves in the kiddy graph I produced months ago which was not taken seriously. There are still no explanations for the massive drop in patrolling / rise in backlog in mid 2016 which was what caused me to begin exerting pressure to run ACTRIAL, nor are there any ideas about the odd coincidence of a sharp drop concomitant with my announcement to retire from my self-assumed role of coord of all things NPP. Kudpung กุดผึ้ง (talk) 03:58, 30 August 2017 (UTC)
Kudpung You're welcome! I've added monthly tick marks on the X-axis of the graphs to make it easier to interpret, which adds half-monthly marks in between as well. I chose not to do anything with the Y-axis tick marks because the purpose of the graphs is to expose the overall trend and allowing you to ballpark what the values are. If there's a need for a specific set of values they can always be found in the TSV files I linked to earlier.
From what I've learned, the massive drop in patrolling in 2016 is due to SwisterTwister no longer patrolling (ref the WMF report from earlier this year), and the remaining NPPers not compensating for that. As for why they didn't patrol more, you'd have to ask the patrollers. Similarly, I do not know why there's a drop coinciding with your announcement. Cheers, Nettrom (talk) 14:33, 30 August 2017 (UTC)

User:SoWhy claims to have extracted data that shows a big uptick in 2017 in the number of pages moved to Draft from mainspace (if I read their post correctly). It's a softer gentler solution than deletion for problematic pages and now supported by a neat toolbar gadget. The return of such pages after being fixed may also partly explain the increase in Draft promotions in 2017. Legacypac (talk) 05:24, 30 August 2017 (UTC)

Basically, yes, although the conclusion was a simple comparision of all such moves since draftspace was created in 2013 (25,000+) to such moves in 2017 (20,000+). I linked the SQL queries I used for anyone interested. Should be easy to create a graph based on them. Regards SoWhy 06:10, 30 August 2017 (UTC)
  • Nettrom, why don't these numbers of active reviewers match with the Community Tech Bot numbers as logged at Wikipedia:Database reports/Top new article reviewers? I'm assuming because it isn't taking Twinkle actions into account, which Community Tech Bot does. Since Twinkle is still the default patrolling tool used by many reviewers, it would be an oversight not to track those actions. I'm also assuming some of the other graphs might be impacted by this as well. TonyBallioni (talk) 17:54, 30 August 2017 (UTC)
TonyBallioni: Thanks for noticing and asking about that! Your assumption turns out to be correct. I didn't see Twinkle mentioned in the documentation and discussion I've read around NPP (e.g. WP:NPP only describes the Page Curation tool). I also interpreted how WP:NPP mentions that certain tools should not be used as "thou shalt use Page Curation", meaning I didn't look into other approaches.
I've found the specific queries that Community Tech Bot uses and will grab updated data that also takes Twinkle patrols into account. Thanks again for noticing this so I can fix it! Cheers, Nettrom (talk) 19:05, 30 August 2017 (UTC)
Heh. The NPP tutorial does aim at getting people to use the page curation tool, because it is the official media-wiki extension for the work (and while it has some kinks, it is quite good in many regards.) But for example: I always use Twinkle for AfD nominations and G12 nominations. Most people prefer the Twinkle CSD log to the Page Curation one. I can see some confusion here, which is why I thought I should point it out. TonyBallioni (talk) 19:19, 30 August 2017 (UTC)
TonyBallioni: Yeah, I can totally understand the use of specific tools to do certain tasks more efficiently, as you mention. And again, I appreciate that you noticed the issue so I could go get some better data. I've updated the graphs, but run into some new challenges that I'll need more time to figure out. Now that other patrol actions are taken into account, we also seem to capture a lot of patrol activity that is not necessarily NPP. For example, the sort name redirect Wood, Charles, created and patrolled at the same time. I filter out those types of page creations in my article creation dataset. Will have to figure out how to filter those out from the other datasets as well, currently they inflate the number of patrol actions and active patrollers up until April 2016, it looks like. Once I have more I'll report back, but it might be a few days. Cheers, Nettrom (talk) 21:22, 30 August 2017 (UTC)
The tutorial was 100% correctly interpreted by Nettrom. It deliberately aims at getting people to use the page curation tool because that's how NPP is supposed to work, and with a group of qualified users. As a purpose designed tool, Curation/New Pages Feed is clearly far superior to Twinkle for processing new pages but the people who don't use it are a) users who have been patrolling since before it was developed and can't break with the habit of using Twinkle, b) new, inexperienced users who don't have access to it because we are trying to limit the damage they do, and C) the WMF declines to make the minor tweaks to the software which will address, among other things, TonyBallioni's reasons for not using it..
Filters, ORES and other AI can help with some of this, but these are very long term developments will and they can never replace the human element. These features are urgently needed. Kudpung กุดผึ้ง (talk) 22:23, 30 August 2017 (UTC)
Number of active patrollers per day: What happened on that day in 2016 when the number of patrollers fell by 86% and stayed there ever since? Is this a flaw in the data? Am I missing something so obvious I can't see it? Is this what Nettrom is investigating now?
It's interesting to note that despite fears and concerns voiced at the time , the introduction of the new New Page Reviewer group does not even show on the graphs, and if anything, the amount of patrolling seems to have increased (ever so slightly). This is important to note. Kudpung กุดผึ้ง (talk) 22:23, 30 August 2017 (UTC)
Kudpung: The data was flawed, it failed to filter out autopatrolled edits properly, which resulted in some weird trends. I've updated the graphs again, and now they're hopefully as correct as possible. The data makes more sense now, and as you can see there's a fairly significant drop in the number of active patrollers with the introduction of the NPR group. It was also present in the previous version of the graph, but less pronounced due to the larger scale of the Y axis. Cheers, Nettrom (talk) 22:35, 30 August 2017 (UTC)
Nettrom, as I have pointed out many times, the significant drop began in mid 2016, which was the very reason I took ACTRIAL of the top shelf in the closet and blew the dust off it. And that's why we're here now. Despite the fact that the number of new articles is constantly falling, where 450 qualified patrollers can't or won't do anything much about the actual backlog, the top 20% of them who are doing 80% of the work have had a positive impact on the system. All we need to do now is to discourage (or even prevent) new uses from trying to patrol new pages and get them to concentrate their efforts on anti-vandalism and recent changes.
You made an interesting observation about the creation of redirects. This is something we need to look at closely because this is precisely how paid editors work to create pages by avoiding scrutiny, and extorting money from BLP subjects. Kudpung กุดผึ้ง (talk) 22:59, 30 August 2017 (UTC)
Kudpung: I am not sure which drop you are referring to when you say that it "began in mid 2016"? The drop in patrol actions in May 2016 is correlated with the reduction in patrols by SwisterTwister, as I discussed above. When it comes to the number of active patrollers, there's a drop in Q1 2016 before participation increases until the introduction of the NPR right in November the same year. Is there some other data I should know about to make sense of this? Regards, Nettrom (talk) 01:33, 31 August 2017 (UTC)

Another update of the underlying dataset resulted in an update of three of the original graphs, and two of the graphs showing recent data with trends. I found a more subtle issue with counting patrol actions in that some patrols appear to have been counted twice. These have now been removed from the underlying dataset and the graphs updated. The graphs that were affected by this were proportion of patrol actions to articles, and the proportion of total patrol actions done by the 25% most active patrollers. Let me know if there are any questions. Cheers, Nettrom (talk) 17:33, 31 August 2017 (UTC)

Deadline in 5 DAYS

DannyH (WMF). There have been no reassuring comments for a while. Could you please confirm that the WMF is on schedule for the 7th? Thanks. (FYI: TonyBallioni). Kudpung กุดผึ้ง (talk) 21:55, 1 September 2017 (UTC)

Hi Kudpung: Yes, the security review for the extension was completed today -- you can check out this ticket. I believe that's the only obstacle we needed to get through. Morten's ready, I'm ready. Kaldari's away right now, but I expect he'll confirm when he gets back on Tuesday. We're on track for September 7th. -- DannyH (WMF) (talk) 22:56, 1 September 2017 (UTC)
Thanks. Kudpung กุดผึ้ง (talk) 23:04, 1 September 2017 (UTC)
@Kudpung and TonyBallioni: It looks like there is still no consensus on whether to send newbies to a special landing page or directly to the Article wizard. We would like to get that issue settled before launching ACTRIAL, as it would cause a lot of chaos to change it in the middle of the trial. Should I convert that section into an RfC to get more opinions? Also, who is going to be in charge of the changes to the last page of the Article Wizard? @MusicAnimal: Is that something you could do? Kaldari (talk) 17:26, 5 September 2017 (UTC)
@MusikAnimal: Oops, ping fail. Kaldari (talk) 17:28, 5 September 2017 (UTC)
Sure. I think we just need to add some user rights CSS classes around each option — MusikAnimal talk 17:38, 5 September 2017 (UTC)
This is pretty exciting stuff, like we're all on our way to see The Oracle :) — fortunavelut luna 18:21, 5 September 2017 (UTC)

Countdown

I think the countdown clock posted at the top is misleading. It looks like we're ready to go on September 7th, but we haven't set a specific time for it. I don't want people to think we're promising to hit a specific second. -- DannyH (WMF) (talk) 18:31, 5 September 2017 (UTC)

Is the clock suggesting 00:00 UTC? Kudpung will have to get up at 7AM  ;) — fortunavelut luna 18:34, 5 September 2017 (UTC)
@DannyH (WMF) and Kaldari: the current time was set by Kaldari with this edit today. I think Kudpung took it from the checklist page, where There'sNoTime set it to 00:00. I've updated it on that page based on Kaldari's time. TonyBallioni (talk) 18:36, 5 September 2017 (UTC)
Also, please note that it isn't possible for us to give an exact time. A 1 to 2 hour window is about as accurate as we can get since deploying new software is a complicated process. Kaldari (talk) 19:10, 5 September 2017 (UTC)
Kudpung won't even be around. He's a volunteer and has other stuff in RL to do. Kudpung กุดผึ้ง (talk) 22:57, 5 September 2017 (UTC)
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:Autoconfirmed_article_creation_trial/Archive_3&oldid=802599482"