Jump to content

Wikipedia talk:The future of NPP and AfC/Archive 4

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Archive 1Archive 2Archive 3Archive 4Archive 5Archive 6Archive 7

ACTRIAL as a research experiment

As I mentioned over the weekend, I met today with some WMF folks interested in New Page Patrol & ACTRIAL: Kaldari and MusikAnimal from Community Tech, and Aaron Halfaker and Jonathan Morgan from Research and Data. The people we've been talking to, here and on related pages, have made a lot of good arguments in favor of ACTRIAL. So we're interested in running ACTRIAL as a research experiment, so tmhat we can look at the impact on new user retention and productivity, as well as the impact on page creation and reviewing.

This is a new plan, and we need to run it by a few people -- both at WMF, and here, with you. I'll describe the current outline of the plan, and we're interested to know what you think.

This week, Aaron is going to do a statistical power analysis, which will help us understand how long the trial would have to be, in order to get statistically significant results. We'll share that analysis with you next week, once Aaron's done. We've done similar experiments before on the Articles for Creation workflow and the Wikipedia Adventure, and we've been able to get statistical significance within two to four weeks on those. But this is a different case, and Aaron's analysis will tell us more. We also need to make sure that we've got resources to run the analysis for the experiment; we're working on that now, and we hope to have that settled by next week as well.

Once we've got that settled, our idea is to propose the trial as an RfC, to make sure the community is okay with WMF making the change. As part of that process, we'll put together a set of metrics to evaluate, with your help. (We'll use the ACTRIAL questions as a guide.) Then we'll make the change, which will probably be sending non-autoconfirmed users to the Article wizard to create their article in draft space. Once the trial's over, we'll turn it off, the analyst will crunch numbers, and we'll present the results for everyone's discussion.

I hope this idea comes as good news for everyone here. :) Either way, I'd like to know what you think. -- DannyH (WMF) (talk) 22:14, 26 June 2017 (UTC)

Had to double take when I saw the section heading in my watchlist :) - yes, very unexpected but definitely good news. Thanks for your willingness to listen to us here and across the other talk pages, as well as gathering requested stats etc. jcc (tea and biscuits) 22:20, 26 June 2017 (UTC)
There is already a consensus for WP:ACTRIAL, a 3-6 month trial with 1-3 month review: Wikipedia:Village pump (proposals)/Proposal to require autoconfirmed status in order to create articles. It's not a small one as well: 500 participated with very good support for the trial. Esquivalience (talk) 22:38, 26 June 2017 (UTC)
I agree with Esquivalience. There is already a consensus to run this trial. Unless there is some evidence that consensus has changed while the WMF has delayed implementation, then there is no reason to revalidate the consensus. - MrX 23:05, 26 June 2017 (UTC)
  • First, let me say that despite my clearly being on team ACTRIAL, I do not have the same negative feelings about the WMF as some people on here do. When TNegrin (WMF) posted about this at WT:NPP, I was actually ecstatic and told other users who were questioning it that they should be thankful for the attention and wait and see. During the time since the report has been launched, I am more than grateful to MusikAnimal (WMF), who got us the statistics we needed despite this not being his job and who actually talked with us about what we wanted to know rather than talking at us. The same goes with Kaldari, who has worked with us on technical questions and features and despite what I see as our disagreements of philosphy on ACTRIAL, I believe that he really wanted to engage us. I don't say this because they are longstanding community members, I really haven't interacted with either of them much before this, but because they actually listened to our needs and gave responses that weren't talking points.
    I cannot say the same for DannyH (WMF), who I honestly wish no ill will and to my knowledge I haven't gotten into sparring with during the process beyond disagreement on what would work better (and if I have been uncivil, I am sorry.) Danny gave me the impression this entire time that he was here to sell the community something and that he did not want to listen to our concerns. This might have been unintentional, but it is the feeling I have gotten, and based on other comments people have made here (some which could be construed as rude, others to the point but civil), I feel that he has lost the confidence of the community in this matter. To be straightforward: this proposal smells like something put together when you release a brand new idea to the public, and it fails, so you need to save face. I think that is an accurate description of where we are right now. While I am glad that we are finally at a point of engagement here, I honestly did not feel like I was being heard until this weekend.
    So what does this mean for my views on ACTRIAL and his new suggestions? A few points:
  1. I felt and expressed to others on-Wiki that an RfC would be needed if we were to implement ACTRIAL locally through edit filters or the blacklist. This is mainly because it was my view that we would be going against an advisory view of the foundation, which runs the servers we work on, and that renewing the consensus would be needed if it were to work without foundation staff issuing an office action against it. I was and am prepared to go to a sitewide RfC very quickly if this was the case.
  2. I do not think now that the WMF seems to realize how big a deal this and how much trust they have lost with us over this that an RfC is needed. There was the 2011 consensus, diverse groups of the community have commented in the past that they felt the 2011 consensus still stood, and the WMF now seems to be ready to help implement themselves. This is a very different situation than if we would be implementing against their wishes.
  3. I think an RfC after the trial on whether it is permanent or not makes sense regardless of which route is taken.
  4. If the WMF insists on an RfC to the point where ACTRIAL will not go forward without it, I think it is important that it be crafted by members of the community and not WMF employees, either in a professional or volunteer capacity (they are the same human, and we all have biases). The reason I posted the paragraphs above about the staff personalities is this: there is lost trust, and the best way to regain it if insisting on an RfC would be to let the community be in charge of the drafting.
I hope these views are helpful and are seen not as having animosity towards anyone in particular or the WMF in general, but as an expression of where I feel those engaged in this conversation are right now. TonyBallioni (talk) 00:00, 27 June 2017 (UTC)
We're not here to sell you anything or tell you what to do. The basic reason there are conflicting points of view is because the WMF and New Page Reviewers have different priorities. New Page Reviewers are more concerned with enforcing Wikipedia standards, while the WMF is more concerned with broadening the content of Wikipedia to include under-represented groups and topics (since those groups are our most rapidly growing audiences). These diverging priorities put us on a collision course. Personally, I think that both points of view are valid, and I hope we can keep trying to listen to each other without falling into personal attacks, conspiracy theories, and the like. The WMF has presented its point of view and it's mostly been rejected by the present company. The WMF took that feedback and also looked at the new data that MusikAnimal produced and decided that it would be better to work with the community on ACTRIAL than against them. Surely, that should be appreciated rather than criticized, but I'm not going to waste my breath on that argument. Conducting a new RfC was mostly my idea, as I don't think there would be any harm in collecting broader community feedback before we embark on ACTRIAL. I fully expect such an RfC to pass, but we might get some useful feedback on how to implement it (for example, how should we deal with edit-a-thons?). I'm also slightly worried that if the WMF implements ACTRIAL without a new RfC there will be members of the community that will complain that we are acting unilaterally (as the community is much more skeptical of actions by the WMF than by community admins). If everyone here strongly objects to the idea of a new RfC, we don't have to do it, but I would be surprised if that were the case. How about we collaborate on drafting a new RfC? What do folks think of that idea? Kaldari (talk) 00:56, 27 June 2017 (UTC)
All a wide-scale RfC does is attract the trolls and vandals with no legitimate want to contribute. The old RfC was carefully discussed and crafted. It was not a question of yes or no, it was an ensemble of viewpoints that fully showed each participant's view on the topic. And it was not the participants who stopped the trial, it was the WMF that stopped the trial. ACTRIAL is not new, it is unfinished business that is years behind schedule. I must agree with the above, that the refusal to implement ACTRIAL has shattered any trust that the WMF supports Wikipedia as well as they can, and I have little confidence that this new proposal will materialize in something. Please prove me wrong. Esquivalience (talk) 01:05, 27 June 2017 (UTC)
And I don't want to explain this again, but here goes: the garbage that comes through the new pages feed every day is abysmal. If "broadening the content" means including every dreamer's band and every YouTube unknown with a total of ten views, then nobody would trust Wikipedia anymore. Esquivalience (talk) 01:10, 27 June 2017 (UTC)
(edit conflict) Kaldari, I understand why my statement above could make you feel frustrated and like you can't win. It is good to know that the WMF did not see its role as selling anything, but like I said, even if it was unintentional, that is what it felt like, and perception matters a lot. Like I said above, I am happy that we are finally getting engagement from Danny after a month rather than just being talked at, which is what it felt like. That might not have been the intent, but it came off that way. Putting that out in the open I think is necessary for any future movement here.
Re: RfC, I don't think it necessary now since the WMF seems amicable and we have a previous consensus to implement, I would also be fine going through one if others want it. In terms of collaboration, I very strongly feel that WMF employees should not take part in drafting an RfC either before or after ACTRIAL because of the history here, both from the last six years and the last month. TonyBallioni (talk) 01:17, 27 June 2017 (UTC)
TonyBallioni, I've learned a lot over this past month about this process, and the people here. I knew that the findings in my report were going to be controversial -- that some people would agree, and some would very strongly disagree. I didn't want those ideas to get run over, so I argued for that approach. I probably did that too hard. I'm sorry that I came off like I didn't really care, and I'm sorry that I wasn't listening closely enough. The reason why I'm still here, and why the Community Tech team is still here, is because we care about this workflow, and we care what happens on Wikipedia. -- DannyH (WMF) (talk) 01:31, 27 June 2017 (UTC)
@TonyBallioni: I'm fine with that on one condition: That the WMF be allowed to help design how the ACTRIAL is implemented. We want to make sure that it is implemented in such a way that we can actually get useful, statistically significant data from it. We also want to minimize collateral damage to the project. What we have to offer in return is (limited) development and research capacity. In other words, we can (hopefully) make the ACTRIAL roll-out go smoothly and we can collect and process the data that the community will need to evaluate the effects of the trial. Kaldari (talk) 01:58, 27 June 2017 (UTC)
@Kaldari and DannyH (WMF): thank you both for your responses, and for taking the time to have this dialogue. I really do appreciate it. I can only speak for myself, but I certainly see that as being a positive role for the WMF to play in this. I would also assume that this would be in collaboration with the community so it can truly be a jointly owned venture. Despite what I posted above, I probably have one of the more optimistic views regarding the Foundation, but I think addressing the last month was necessary in moving forward positively.
Let's see how others feel about an RfC: you could count me as a weak oppose on having one, but as in all things, I'm willing to be persuaded otherwise. TonyBallioni (talk) 02:09, 27 June 2017 (UTC)

How about this agreement:

  • The WMF implements ACTRIAL. This means that if a user has fewer than 10 edits and has registered less than 4 days ago, then they cannot create articles. No half-baked solution, no RfC, use the consensus developed at the RfC we already have.
  • No stalling. If this isn't done within, say, 60 days, then I would say that it is safe to say that this is not happening.
  • In return, the WMF gets to collect data however they want, publish it however they want, modify the interfaces as needed, use A/B testing if necessary. However, the WMF does not make the final decision. Instead an RfC is created and if there is a good consensus for ACTRIAL then it remains. Esquivalience (talk) 02:48, 27 June 2017 (UTC)
    • Esquivalience, I understood Kaldari as saying that the WMF wanted a role in the specifics of the implementation (technical, timing, etc.), which I think makes sense. I agree with all of your other points, which is why I am especially opposed to WMF employees (on whatever account) having a role in the drafting of any potential RfC, pre or post. ACTRIAL is first and foremost a community initiative, but working with WMF and the resources they provide I see as being a positive if it is done right, especially if the alternative is a mega-RfC about flipping the switch without them. I don't mind doing that if nothing comes of this, but I'd prefer not to. TonyBallioni (talk) 03:07, 27 June 2017 (UTC)

DannyH (WMF)'s post arrived at 5:14 am, my time while I was already up and checking out some other communications regarding these issues. I decided to mull it over while I had my breakfast and drive to my office. Lo and behold, I started reading a reply to it and I thought "Huh? Did I type that while I was having my breakfast?" (at my age, the short term memory sometimes pays tricks on one). So identical to my own thoughts, it wasn't until I saw Tony's signature on it that I realised I hadn't written it after all.

This news from the Foundation sounds good on the surface but for me it comes with mixed blessings and we need to proceed with caution. I'm not sure we would want the WMF to take over ACTRAL. That was never part of the 2011 project. Wresting it out of the hands of the volunteers gives them the power to manipulate it to their own ends. The last two times they did it, it also involved NPP and while it slowed down developments of our work, it ended in disaster for their reputation, and I was furious that an employee made us wait 5 whole months for a simple request and then went to the Signpost claiming the kudos for all my hard work. Now perhaps people will understand why I for one, am very bitter when dealing with people who are just as evasive as that contractor who under the pretext of having been sent to give us a hand, turned to be a Foundation mole and was feeding us with fake delays and disinformation. What WMF need to understand is that their 'offer' now comes a bit late, the damage is done and Danny's department has already lost any of the trust and confidence that the community might have had - exacerbated by the fact that they refuse steadfastly to tell us who their superiors are and what was going on with Wes Moran with whom I shared an hour's Skype with Kaldari. Don't get me wrong, Ryan is one of the good guys, but obviously he's between the horns of a dilemma (pun intended).

FWIW, if there is going to be another RfC, while most people, as I do, consider the original, very strong consensus to be still valid, a new RfC has been drafted and is waiting in the wings in case it's needed. No changes are needed to anything that was proposed at ACTRIAL, or the templates and interface messages that were already prepared directly on that page for all to see at WP:ACTRIAL. What I would appreciate is that the WMF would climb down from centre stage and finally understand that among the thousands of volunteers there are plenty of people, if not more, who are just as qualified as any of the paid staff - if not more so. Unfortunately there is this attitude, which I have even encountered in RL at several Wikimanias over the years, even from Howie Fung, that because they are paid, the employees certainly assume they are better and know best.

I would warmly welcome a collaboration between en.Wiki and the WMF on the development of the new landing page if we could be treated as equals, and in equal numbers around the designing table. I would like to see a continuation of the 1,000 lines of code that have already been written, but that depends both on the WMF team and whomever the community sends as its delegates. Kudpung กุดผึ้ง (talk) 05:25, 27 June 2017 (UTC)

Kudpung, I can tell you who our superiors are -- Ryan and I run the Community Tech team together, and our boss is Toby Negrin, the interim VP of Product. Toby's boss is Katherine Maher, the ED. This information is on the Wikimedia Foundation staff page. Wes wrote to the Wikimedia-L mailing list about why he left, which you can see here.
Ryan can probably talk about the call with Wes; my guess is that Ryan wanted Wes to know about the NPP problems, and they didn't follow up because Wes was busy with other things. Ryan, Leon and I work together, and we met with the researchers today so that we could figure out how to implement ACTRIAL as a research project; we made the decisions together, as Ryan says above. You can check any of these facts by asking anybody else who's involved. Does that help? -- DannyH (WMF) (talk) 06:01, 27 June 2017 (UTC)
Personally, I have no idea why Wes left. I could only speculate. Also, can we please lighten up on the personal jabs? Danny's offering an olive branch here and getting drawn and quartered in return. We're all tired and frustrated at this point, but I think the best solution for the future of Wikipedia is for us to figure out how we can cooperate (as Kudpung has asked us to do many times). We don't want to "take over" ACTRIAL, we want to help design how the ACTRIAL is implemented and help with the relevant data collection (with what little resources we have at our disposal). Hopefully that can be a win-win situation. Kaldari (talk) 07:20, 27 June 2017 (UTC)
It does, DannyH (WMF), and thank you. The staff page does not however depict any organigram or arborescence. I always liked to think of the WMF as a relatively flat hierarchy on the German model of the 70s and 80s. Perhaps it basically is, but the main difference is that in the Wikimedia family of projects the volunteers - the real workers - don't have any say in the product, the management, or on who gets hired and fired. Moran's valedictory, which tells us nothing, is a carbon copy of all the others, except perhaps for my good friend Philippe who actually also thanked the volunteer community for their work. Just like I spar with you, I used to fight with Jorm. We ended up being good friends - on and off Wiki. Let's see. Kudpung กุดผึ้ง (talk) 07:27, 27 June 2017 (UTC)
Thanks, Kudpung. I think there are ways that the community influences WMF management, but it's not a direct line from one individual to the top of the agenda. There are community members who are voted onto the Board, and they actually have direct impact on hiring and upper management, as part of that group. There's an annual planning process that gets evaluated by the community-run Funds Dissemination Committee. There's currently a year-long Movement Strategy process that includes a lot of community input. And there's the Community Wishlist Survey -- you've said before that you're not impressed with it, but it's a method for getting direct input on the product ideas that people think are most worthwhile. And then there's us talking to you right now. :)
But all of that is cold comfort when there's something that you know is super important, and you've been trying to raise it to management's attention for years, and you still can't manage to get anybody to work on the problem. When you talked to Wes -- the VP of Product, the guy who could actually tell a product team to work on this problem -- I expect it felt like someone's finally paying attention, and then he disappears and nothing happens. It would make me wonder if there was a plot to thwart the community, that maybe Wes was fired because he was interested in doing what the community wants. I don't know if that's exactly how it felt to you, but obviously it was disappointing and frustrating.
I honestly don't know anything about Wes' departure besides that post on Wikimedia-L; I wasn't close to him, and we didn't talk. Here's what I can tell you about the transition from Wes to Toby: Wes never talked to me about NPP or Page Curation. He might have talked to the Editing/Collaboration team, but I didn't hear about it. Since Toby stepped in as the new interim VP, he's been looking at problems like this, and he's asking why we haven't been working with the community on them. Last month, he brought people from several teams together and said that he wanted a team to step up and be more proactive, if "proactive" is a word that applies when it's six years later. He thought Community Tech was the best team to work on it, and we wanted to work on it. We all knew the current limitations on our resources, but we wanted to start engaging with this group and this problem. So the transition from Wes to Toby isn't a sudden roadblock. It's progress.
Kudpung, there may be some things that you and I will disagree about until the day that we die, and that's fine -- it's productive and interesting to work with people who have a different point of view. I would like to move beyond the personal jabs, as Ryan mentioned -- if we're going to spar, then let's spar about the ideas. I'm good with that, if you are. -- DannyH (WMF) (talk) 21:47, 27 June 2017 (UTC)
  • I'm not ok with this because it's based upon autoconfirmed status. That introduces an arbitrary delay of four days for no good reason and that seems especially disruptive for outreach events. For example, in July, I shall be attending the Oxford Symposium on Food and Cookery. This year it's a three day event and the Wikithon organised by the British Library is just one day in that. The event material says "This workshop will provide you with all the skills and resources necessary to create new entries ...No prior experience is required". Introducing a four-day delay is an obvious blocker for such activity. And Wikipedia is not the only game in town. For example, another event at the symposium is about adding new material to YouTube. I might try that myself as I've not done this yet and video seems an intimidating format for a newbie. Does YouTube tell you to go away and make 10 comments to other videos before you can upload one of your own? Does YouTube tell you that you have to wait four days simply for time to pass? I shall be surprised if it's so hostile to new contributors because my experience of other platforms such as Twitter is that they are welcoming and eager for people to participate. If Wikipedia heads off in the other direction then those other platforms will just eat our lunch. Andrew D. (talk) 06:56, 27 June 2017 (UTC)
Andrew, you're just being deceptive again. Here in Thailand I'm almost as active on off-Wiki projects as you are in the UK, and you know full well that for editathons there are special features available that you only have to ask for. There are no reasons why editathon participants would be disadvantaged or even discouraged by what's going to happen when ACTRIAL is rolled out, and you can ask Doug if you don't believe me. You do a lot of good work out there, don't spoil it by comparing Wikipedia to social media. Kudpung กุดผึ้ง (talk) 07:44, 27 June 2017 (UTC)
@Kudpung: Unless we're going to grant confirmed to all the new accounts for an editathon, they would be hit by any edit filter. However, there's nothing bad about sending all the new articles created by editathon participants through AfC. If whoever is running the editathon is competent, they can easily spin the review process as a positive to "give you feedback on your new entries" etc etc. (Made especially easy because it actually is positive). ~ Rob13Talk 10:36, 27 June 2017 (UTC)
For editathons there are special features available that you only have to ask for. There are no reasons why editathon participants would be disadvantaged or even discouraged by what's going to happen when ACTRIAL is rolled out. Editathon facilitators know well in advance when they have an upcoming event, which leaves them plenty of time to have an admin on call in real time. User:RexxS knows about theses things - why doesn't Davidson? Kudpung กุดผึ้ง (talk) 10:42, 27 June 2017 (UTC)
@Kudpung: The "special things" are WP:Account creator, mostly. We have no way of bypassing an edit filter that checks for autoconfirmed or confirmed except by granting confirmed flags, which I don't think is a likely community practice. There's no good reason why editathon participants should bypass AfC, though. Editathon coordinators could, of course, have admins or AfC reviewers on hand to provide instant feedback on articles. ~ Rob13Talk 10:51, 27 June 2017 (UTC)
Rob, I know what the special things are, and I've run a dozen editathons. Of course pushing them all through AfC would be the 'easiest' solution, but if you've ever facilitated an editathon, you will have noticed that your audience are not your usual vandal children, spammers, and hoaxers you get at NPP. The idea (or at least the way I train them) is to get them to understand enough that by the time they go home they've done 10 edits and just need to wait another three days. They will have been taught enough to not necessarily need to go through the wizard or AfC. Kudpung กุดผึ้ง (talk) 11:00, 27 June 2017 (UTC)
Agree with Rob that there is no good reason for editsthon participants to bypass AfC. The other solution would be to debundle granting confirmed status to account creators if this is a big enough deal. I see these types of things to be things we would work through with the WMF during the implementation proxsss. TonyBallioni (talk) 12:22, 27 June 2017 (UTC)
I've been watching the proposals with interest, and been trying to formulate what impact ACTRIAL might have on training sessions and editathons where I'm involved. I think that any problems that might arise would have fairly straightforward solutions. First of all, I encourage all first-time editors to work in their sandbox to begin with, as it allows them to test out possible amendments to articles as well as simulating new article creation. I would always encourage expanding existing articles over new article creation for new editors anyway, as the required skill set is greater for the latter activity. I'd be perfectly happy to see those editors who seem capable of article creation doing so in Draft: space, as the guidance from AfC is valuable. It would be nice to have the ability to confirm editors as part of AccountCreator, but it's not essential. If ACTRIAL were in place, I'm pretty sure that I could simply create a page in the appropriate space for the rare few who learn so quickly that they are ready to create an article before they are autoconfirmed. It would not be a great burden during the course of an editathon. I also doubt that I would be taking a serious risk with the integrity of our project, as I make a point of getting to know participants, so should be in a position to judge when they are ready better than an arbitrary "10-edits & 4-days" restriction would do.
TL;DR: I'd be quite happy with ACTRIAL and don't expect it present any real challenges to any reasonably well-planned training session or editathon. Cheers --RexxS (talk) 12:50, 27 June 2017 (UTC)
I agree, RexxS. My experience as a mentor for potential article creators/editors is minuscule compared to most who are commenting here, but experience in various other teaching disciplines has demonstrated to me that newbies and students actually appreciate PAGs and welcome a bit of mentoring. I liken it to teaching someone how to swim before you strap a SCUBA tank to their back, and that it's much better to start them at the shallow end of the pool. The biggest obstacles I've seen in newbies who truly want to contribute and create quality articles is a lack of self-confidence. What if I mess up? A stop-check (user sandbox), how-to effort (mentoring) will address such issues. I'd much rather collaborate with new editors in a friendly mentor program than be forced to explain why their article is crap and up for deletion at AfD, the latter being a move that is proven to raise a lot of hackles and discourage potential editors. We should also keep in mind that while many on the NPP team feel that we're fighting a losing battle, we must not forget the other groups of editors we're sending those articles to for clean-up or deletion, including admins, copyeditors, Wiki gnomes and various project teams. I'm of the mind that if there's a snake in the bullpen, just kill it - don't waste time and energy appointing a committee on snakes, but that's another story for a later discussion. Bottomline - WP will continue to be the encyclopedia anyone can edit. It's human nature to treat free and easy with less value than something they acquired through time and effort. The symptoms of WP's unsustainable development tell us it's a growing concern that comes with consequences, including relentless vandalism, low morale and lost incentive which creates problems with editor retention, and the growth of poor quality articles and promotionalism, all of which threaten the integrity of the encyclopedia. My antenna may not pick up all channels, but the benefits of ACTRIAL come in loud and clear. Atsme📞📧 16:21, 27 June 2017 (UTC)

Update

Hi everyone, here's an update on the ACTRIAL experiment.

Last week, I said that we'd get a statistical power analysis from Aaron to help us understand how quickly we could get statistically significant results, and here it is: Research talk:Autoconfirmed trial experiment.

Aaron says that we could reach statistical significance in a week, or two weeks to be sure. I think that's too short to really judge impact on the backlog and the NPP system. I would suggest four weeks, which I think is enough to give people time to adjust, and see how it feels. I think going much longer than that would have diminishing returns, and I'm eager to actually see the findings, which a longer test period would delay. I'm curious to know what you all think.

Currently, we're working on securing extra funding for a data analyst contractor. All of the researchers on staff are working on other projects, but there's a contractor that we're hoping to work with who's done a lot of Wikipedia-related research. I'm not sure how long it'll take to get the funding approval, hopefully no more than a few weeks, and I'll keep you posted on how it's going.

Once we've got the analyst on board, then we'll come up with a specific proposal for the metrics that we're evaluating in this trial. We'll use the list of questions on the ACTRIAL page as a guide, and we'll post the proposal for discussion.

So that's where we are right now; let me know what you think. DannyH (WMF) (talk) 17:53, 3 July 2017 (UTC)

We fixed the duration at six months to provide a sample period that would cover seasonal variations. We must not forget, for example, that a large number of inappropriate pages are made by children and that not all English language regions follow the same academic calendar and national holidays as the USA. There are also other events that affect the creation of new pages such as items of international news. The period was determined by community consensus. Funding approval + the work: what kind of time scale are we looking at? Another year? When is the collaborative discussion over the design and function of the landing page going to begin? Is there any reason why the conceptual (i.e. look and function) design cannot be discussed right now? Kudpung กุดผึ้ง (talk) 09:28, 4 July 2017 (UTC)
A longer trial period would better cover the likelihood that as time went on, less sophisticated COI editors would "get wise" to the change, and advice pages on the internet on the lines of "how to get yourself/your business on WP" could be adapted to help get round the new barrier: Noyster (talk), 10:24, 4 July 2017 (UTC)
  • I echo Kudpungs thoughts as to when the process for designing the implementation can begin. I think this can and should go on simultaneously as the funding approval and hiring process for a data analyst. There is a lot that can be done without that position on board. I think 6 months would be a long enough trial period to get sufficient data. TonyBallioni (talk) 10:34, 4 July 2017 (UTC)
The approval should be no longer than a few weeks. We've already got the contractor who's willing to do the work, and he's been working on Wikipedia-related research for the WMF before, so there won't be a protracted hiring/onboarding period.
We'll definitely talk soon about the design and the mechanics of how it'll work -- I just haven't had a chance to get together with Kaldari and MusikAnimal to figure out what's possible. We're going to talk on Thursday, and I'll be able to post more about that on Friday.
About the time -- there's a couple reasons why I think six months is too long. For one thing, we're going to have the contractor for three months, to set up the experiment, run it and do the analysis. Also, if it turns out that this change is not actually good for the wiki -- we get more spam, new editors burst into flame, the house burns down -- then six months is a really long time before we figure that out. I'd like the analyst to be part of the conversation around this. DannyH (WMF) (talk) 17:24, 4 July 2017 (UTC)
Why is a contractor required to do analysis? Why not reveal the data and let the volunteers analyse it? Is this not a community run project? --SmokeyJoe (talk) 12:47, 6 July 2017 (UTC)
I concur. It's likely that there is more expertise in the community than the foundation is capable of hiring. We have CS and maths professors here, among a plethora of other technical professionals. I hope that the raw data will be released in addition to any analysis performed by the WMF-hired person. Rentier (talk) 13:17, 6 July 2017 (UTC)
If any volunteers want to help analyze the data, I'd be happy to point them in the right direction. I expect all the data will be publicly available. Kaldari (talk) 19:09, 6 July 2017 (UTC)
A couple of elephants in the room here: 1/ After all this is there really and suddenly a change of WMF sentiment WRT ACTRIAL? 2/ The subtext of this negotiation over trial duration is, do we want to risk making it long enough that it becomes permanent due to inertia? ~Kvng (talk) 18:02, 4 July 2017 (UTC)
It cannot become permanent due to inertia because the terms of the consensus were quite specific that that it return to 'normal' after six months. That was to be followed by a further one month of monitoring to compare withe pre trial statistics and to evaluate the trial itself. If it were found that the trial proves to be a net benefit, new restriction would be rolled out as a permanent feature.
My question regarding talks on the development of the new landing page - which is equally important - have not been answered. Is there a reason why this has been ignored? Kudpung กุดผึ้ง (talk) 20:10, 4 July 2017 (UTC)
Sorry, that was what I meant about "the design and mechanics". I think we can redirect non-autoconfirmed people to the Wikipedia:Article wizard, but I want to talk to Kaldari and MusikAnimal about it. We'll talk Thursday, and then I'll be able to talk about it here on Friday. DannyH (WMF) (talk) 20:38, 4 July 2017 (UTC)
@Kudpung: So you really just want to do a trial to collect data and then I guess move into another phase where we try to get consensus about what to do permanently. In the spirit of Wikipedia, I would prefer a WP:BOLDER approach. ~Kvng (talk) 23:23, 4 July 2017 (UTC)
@Kvng:, I am the strongest and most vociferous proponent for ACTRIAl, and to have it rolled out as a permanent feature as soon as possible. So much so, that I have even been sent totally inadmissible emails from the Wikimedia Foundation telling me to lay off. Please don't fall into the classic Wikipedia habit of taking tings put of context. There has been so much criticism over my efforts of late that I am already looikg round for another hobby, and of course that would also be playing right into the hands of the Foundation and my other detractors, but that's the way it will be. Kudpung กุดผึ้ง (talk) 03:23, 5 July 2017 (UTC)
  • I've been one of the more vocal proponents of ACTRIAL recently, and I would not support a permanent imposition without a new RfC (and ideally one fairly quickly after the trial). I fully anticipate that I would support it being made permanent: I don't see any reason to suspect this being different than Jimmy's fiat that IPs not be allowed to make pages, but I'm also personally not confident enough in my ability to predict the future that I'd be willing to say 100% that the comparison's raised to Citizendium might not come true. I don't think they will, but I'd prefer being morally certain of it.
    DannyH (WMF). re: the timeframe, I get from the WMF's standpoint the desire to have the analyst on board while the post-ACTRIAL discussion is ongoing. In terms of the budget request would it be possible to request the funding for the contractor to be split? One month designing the questions at the beginning and then two months after ACTRIAL for analysis? I get the personnel reasons for wanting to stick it together, but one of the key values of contractors over employees is that you can bring them on and off as you want. I think Kudpung's point about needing a time frame along the lines of six months because of variations within calendars, etc. is important, and I also think it is important because we've had a large dip in the backlog before, and then it quickly shot up to 22k. My suspicion is that this is often caused by adding a particularly active reviewer or two who then burn out. A longer trial would help control for this and other factors. TonyBallioni (talk) 02:02, 5 July 2017 (UTC)
I don't think it's so much to do with burn out as that it's simply a horrible job. It's like doing community penance cleaning sewers instead of going to jail; it's worse than boot camp in the British army in the 70s. If you're not wallowing in written excrement you're being beaten over the head by the trolls and spammers whose articles you've tagged for deletion. Most of the users who ask for the bit at PERM haven't got a clue what they're letting themselves in for, while others just avoid any drama by taking the uncontentious low-hanging fruit. That's where the backlogs come from. Kudpung กุดผึ้ง (talk) 08:12, 5 July 2017 (UTC)
Don't forget the opportunity to be vilified at PICKYOURLETTERS, supervised by editors who don't believe in deleting any thing and being forced to keep up with the rule changes made by editors who have no experience patrolling new pages. Legacypac (talk) 09:01, 5 July 2017 (UTC)
Whatever the cause of it, the return to the median of a growing backlog is something we have experienced before. I'm glad its going down now, but I'd prefer the data be gathered over a longer period of time to help account for activity spikes, etc. TonyBallioni (talk) 15:06, 5 July 2017 (UTC)
TonyBallioni: When the funding for the contractor gets approved, they'll be able to guide the design of the experiment. I want that process to be as transparent on-wiki as possible, and include the concerns that people have brought up here. Looking back a couple days, I probably shouldn't have jumped into making suggestions about the time period :) -- that's actually part of what we're hiring somebody to help us all figure out. Sorry about that, I get ahead of myself sometimes. DannyH (WMF) (talk) 17:40, 5 July 2017 (UTC)
Also, DannyH (WMF): you seem to be under the impression that no analysis of the effects of ACTRIAL can be undertaken until the trial is over. Why can't analysis take place concurrently (at least, after a week or two have elapsed)? If "new editors burst into flame" or "the house burns down" as a result of ACTRIAL, it should be rather easy to see that it's happening while the trial is taking place, at which point a discussion about whether to stop the trial early can take place. ‑Scottywong| verbalize _ 07:12, 6 July 2017 (UTC)
I'm also worried about the time frame for getting the stats. The last time we asked the WMF for some help with NPP stats, the contractor who was foisted on us by the Foundation fed us a load of BS about delays until full FIVE MONTHS later he published his own doctored report and went to Signpost with the false claim 'what a good little boy am I, the volunteer community was wrong' (only to be desysoped shortly after for some other inappropriate action). Kudpung กุดผึ้ง (talk) 11:51, 6 July 2017 (UTC)
When could we expect to actually get underway? I'm a bit concerned that we may end up coinciding with a surge of newbies as the US academic year starts, which may skew the results - or is that not a significant factor? Roger (Dodger67) (talk) 12:31, 6 July 2017 (UTC)
I think if we're waiting a month to get funding approved that's likely. A longer time period would likely help account for this and balance it out. TonyBallioni (talk) 12:34, 6 July 2017 (UTC)
It's amazing how short the WMF's memory is of their last $125,000 research into NPP. Kudpung กุดผึ้ง (talk) 14:06, 6 July 2017 (UTC)
See also this and I’m very much afraid of a repeat of this NPP discussion which was nothing other than deliberate stalling by the WMF, a usurpation of a community project, and a doctored set of stats that were of no use to anyone once that contractor released them to the community. And by rhe way, talking about stalling - we need to get this trial up and runing befire Europe and North America goes on holiday - there are going to be millions of children with nothing better to do than play their favouririte MMORPG: Wikipedia. Kudpung กุดผึ้ง (talk) 14:32, 6 July 2017 (UTC)

Update break

Whatever analysis we do, we'll definitely have to control for seasonal variation. The best way to do that will probably be to compare with data for the same period last year (or across several years). The seasonal variation issue doesn't seem like a good reason to make it last for 3–6 months, but I guess we'll see what the analyst thinks. Kaldari (talk) 20:00, 6 July 2017 (UTC)

A six-month trial is what the community agreed on. I'll repeat again that this needs to be up and running before the Western world releases its children to write attack pages about their teachers and about their favourite non notable video games. We don't want to wait another year for this seasonal disruption to Wikipedia to take place before rolling out ACTRIAL. The trial should begin asap - we can argue about the stats later, the edits won't go away. Kudpung กุดผึ้ง (talk) 03:27, 7 July 2017 (UTC)
@Kudpung: As long as the one month is during a seasonal high point (like early Fall or Spring), and doesn't conflict with a major article-writing campaign (like Art+Feminism), there is no reason to be so disruptive as to prevent the good quarter of content from being actually created, and forcing undo burden on the communities directly effected by the trial. I regularly interact with folks who work on Editathons and other off-wiki programs, and the Draft space and AFC are known as places where new content goes to disappear: which largely defeats the purpose of these off-wiki events. If we run this for more than a month or two, we risk creating a huge disruption of on-going work (such as the education program's content from student editors who often write large-high quality drafts, with limited number of edits and publish them in October/November), forcing workflows to adapt to a workflow that is not going to be reliable or permanent. The one month allows for sufficient analysis that we can do some deeper thinking about the solution, and if we can create better community workflows in response to the change. Sadads (talk) 13:05, 7 July 2017 (UTC)
@Sadads:, I think you misunderstand me. The community has decided that the trial will take place over a sample period of six months. Development of the article creation workflow landing page that should have /could have been developed 6 years ago can restart right now - it's only a few hours of coding.
The purpose of these current discussions is to obtain some technical assistance from the Foundation, not to re-debate the issue of te trial or its consequences. The events you mention will be captured by the trial, which is exactly what we want in order to get some representative data. Anything else would defeat the purpose of the trial. It is not helpful to presuppose its effects with 'what if' and 'we risk'. This is Wikipedia's Manhattan project. Kudpung กุดผึ้ง (talk) 14:00, 7 July 2017 (UTC)
Sadads, you also seem to have a predetermined view of the outcome of the trial, and are assuming that it won't be permanent after it is run. While that is certainly a possibility, and I am open to it, it is in fact probably more likely than not that the community will in fact ratify this as a permanent change unless to borrow a phrase, the wiki bursts into flames. There are already some experienced editors and admins who think that Jimmy's fiat against IP article creation also applies to non-confirmed accounts, and many people want to do away with the TRIAL part of ACTRIAL. As described above, this is actually not that likely to have much of an impact on editathons and if anything gives the participants A six month trial is actually beneficial to the groups you mentioned should the change go permanent because it will give us more off-wiki data to work with in the followup RfC, and could help us adapt the proposal for the eventual permanent implementation and make it more successful for all involved. Additionally a one month trial would not be sufficient for most people on this Wiki who aren't involved with new pages to realize that a change had even happened. The data part of the trial is only one piece: the other piece is giving the community time to see if it likes it. TonyBallioni (talk) 14:32, 7 July 2017 (UTC)
@Kudpung and TonyBallioni: That consensus is six years old, and the timeframe was decided without the researchers on hand to help identify an effective and statistically appropriate means of evaluating the trial. This ought to be a new conversation, building on the new data and a new understanding of what resources will be available. 6 months is extremely, extremely disruptive unless the outcomes are absolutely clear that the limitation will stay in place (which the whole point of the trial, is that it is in doubt -- moreover, we may decide that we need to do something different than AFC/Draft space as a triaging environment: both of which don't appear to work very well for new, limited-time contributors). Moreover, the whole reason that this conversation restarted (the massive backlog at NPP) seems to be abating, through attention and recruiting. Why force a long-term, non-temporary (6 months is not a temporary window of time) process change, when its not necessary for effective evaluation? Let the research experts put forward a plan, based on effective research methods that don't disrupt the community -- its reasonable that if WMF invest resources, it needs to be on terms that match that investment without causing huge overhead for other parts of the community. Sadads (talk) 15:13, 7 July 2017 (UTC)
Sadads, I have been wondering when you or one of your other WMF colleagues would try those arguments. These are just more deliberate delaying tactics, why disrupt the community with such comments?. The backlog is not abating. It has shown a very slight dip just recently, but that is no reason to assume success; if you were familiar with these discussions, you would have seen the graph I prepared showing the backlog over the past 12 months - which incidentally none of your experts has been able to explain. One odd coincidence is that it rose very sharply again in February when I announced that after all these years I was retiring from my 6 years of micromanaging the NPP system.
You are forgetting, or are not aware of the main point: the only reason ACTRIAL was not rolled out in 2011 was that in my and Scottywong's and The Blade of the Northern Lights naïvety at the time, we thought it needed a MediaWiki intervention to which we do not have access. We now know that this can easily be done by a local script (in fact several of them are possible), and we can (and probably wiil) roll this out with or without the Foundation's help over the statistics. New Page Reviewing is in crisis, the quality of our encycolpedia is dropping, the Wikimedia movement is getting very bad press, and we are going to do something about it. Kudpung กุดผึ้ง (talk) 16:07, 7 July 2017 (UTC)
The six year old consensus is still valid, absent any evidence to the contrary. This should have been implemented six years ago and the only things that have changed is that the situation is more urgent and WMF has a much larger budget. There are many good reasons expressed across multiple pages about why this should move forward promptly, and as originally designed. We also need attention to much-needed enhancements to the tools to make the reviewing process more efficient. The recent modest gains on the backlog have been achieved by significant brute force effort from relatively few people. I doubt that its sustainable.
Sadads, could you please explain how a six month trial would be "extremely, extremely disruptive"? It absolutely eludes me that that's the case. - MrX 16:21, 7 July 2017 (UTC)
@Kudpung: As I have mentioned on multiple occassions: I don't work with this particular team, have not been in conversation with them about their goals/tactics/strategy, and to be frank, missed the last few weeks of conversation because I took a vacation away from computers or was focused on other things. I am weighing in based on my experience as a New Page Patroller, long-time editor, and someone who has taught many many new contributors how the projects work.
@MrX:: take for example running editathons or education program assignments: if we suddenly make a policy change for 6 month trial, (where workflow for creating articles move to one system that needs a permission for new editors), and then after the trial we decide to make even more changes to that article creation process (whether making it more strict or trying other technical strategies for making it better for new users). For each of those changes you have a many month lag time for program organizers to figure out what happened. First to figure out that something that has worked for over 10 years, doesn't work any more, and then learn not one, but two major workflow changes. This leads to lots of frustration among many many people not present in this conversation (or even around during ACTRIAL), for a long window of time that is not accompanied by responsive and supportive documentation/strategies that they can learn and rely on for years to come. I would rather we be decisive and holistic after we learn from a trial during one short window of time that can be statistically adjusted for seasonal variation (preferably in the next few months where it won't disrupt as many programs -- which run a lot during the school year in North America/Europe). This allows us to be confident that folks who run these programs, can learn about the change once, and be confident about how it works. Sadads (talk) 16:53, 7 July 2017 (UTC)

RexxS has already addressed the editathon objections at length above. As I mentioned, a longer period would actually give us more points of feedback from these people and help us work with them if it is decided to make it permanent. There is also the pretty obvious answer of them moving the article out of draft space themselves and skipping AfC in four days when they become autoconformed, or asking the person running the editathon to do it for them. TonyBallioni (talk) 16:59, 7 July 2017 (UTC)

Sadads, permit me to remind you that it was your very department that in spite of my advance warning, caused one of the biggest crises for NPP in Wikipedia history (and squandering of tens of thousands of donors' dollars), and that I organised the massive cleanup afterwards by our unpaid volunteers and, like I still do at NPP, put in hundreds of unpaid hours on it. The moral of this story? Each time the paid WMF makes a mess, the community cleans it up for them for free; and that's what we want to avoid happening again here. Kudpung กุดผึ้ง (talk) 17:20, 7 July 2017 (UTC)
Sadads, this has indeed already been addressed and I'm not convinced it's much of a problem anyway. Anyone who is leading an editathon will surely adapt to the slightly different way that new users create will new articles. Meanwhile, thousands of bad article will be prevented from being published and we will be able to evaluate whether fewer spammers and vandals even attempt to create bad articles, and whether the reviewer work load is reduced. Maybe if we can spend less time swatting at the locust swarm, we can actually spend more time helping sincere users create quality articles and become long term editors.- MrX 19:01, 7 July 2017 (UTC)

What needs to happen for ACTRIAL to be a success?

In the interest of being constructive, I think everyone here needs to be a little bit flexible so that we can work together instead of working against each other. WMF is offering an olive branch, and while there is obviously still some anger towards WMF regarding how things went down the last time we tried ACTRIAL, I think we should try to appreciate the olive branch and try to work constructively. While we can technically implement ACTRIAL ourselves, it would arguably be better if we could also enlist the help of some friendly WMF staffers for certain parts of the trial. And, it would be better for everyone if WMF is "on our side" rather than fighting against us. Here are the top things that need to be done to maximize the chances that ACTRIAL is a success:

  1. Review the design of the error messages that will be displayed to non-autoconfirmed users when they attempt to create a new article in mainspace. Are they clear enough? Are they friendly enough? Do they offer enough alternative options for creating the article elsewhere? Do they clearly explain what is required to become autoconfirmed? Do they look good and match the current aesthetic of WP error messages? I created some example messages many years ago, we could use them as a starting point (find them here).
  2. Decide how to handle edit-a-thons and other off-wiki events. Do we need to construct an official system for handing out confirmed status to users participating in these events? Or, do we need to communicate clearly to the organizers of upcoming events that new users will need to create their articles in AfC? What's the most effective way to communicate these changes to the event organizers?
  3. Determine what data we need to obtain in order to measure the success of the trial, and determine how to get that data. This might be something that WMF's data analyst contractor can help with. However, it will be good for the community to provide input on the data we believe to be important. Primarily, the effects we want to measure revolve around the NPP backlog and editor retention, but we'll probably want to get more specific.
  4. Come to an agreement on how long the trial will last, and when it will start. Yes, the original ACTRIAL RfC decided that it would last for 6 months. WMF seems to want to run it for 1 month. Can we meet in the middle and maybe go for 3 months? Running a new RfC on this will delay the implementation of the trial for a long time, and probably won't provide a lot of value for that time lost.

Don't kid yourself, implementing this trial in a way that maximizes its chances for success will take a lot of work and preparation. I think the above 4 points are probably the most important pieces that need to be worked on right now. Does anyone have any other items that need to be addressed? Should we start a new page to coordinate this work? ‑Scottywong| talk _ 21:12, 7 July 2017 (UTC)

Scottywong, thank you for saying these things. I'm of the same mind as you. To respond to your points:
  1. I think something that needs to be worked on is making the error messages more modern/even colourful if possible. If it is a standard WP error message (like when a non-sysop tries to view a deleted revision), I don't think people are going to be as likely to read.
  2. I think the immediate best way to handle it would be to educate editathon coordinators and direct them to have people send things to draft space. As a project we should work on educational materials for the people who run off-wiki events on how best to handle the upcoming changes. I would ultimately like to see the ability to confirm accounts debundeled to the account creator user right, but I think it would be best to include that in the post-trial RfC.
  3. Community input would be ideal, but I do think this is something that should probably be worked on with the contractor.
  4. I was actually just thinking myself that I would be fine with a 3-4 month compromise. I think 6 would be ideal, but compromise makes the world go 'round and figuring out a middle ground is better than fighting.
As to the new page, I have been thinking about that myself. I think creating Wikipedia:Autoconfirmed article creation trial, 2017 will be necessary at some point to make it more clear to everyone what is going on here. I think for now it makes sense to have the basic pre-planning conversations here so as to not create another page for someone to follow. Once we have a basic framework established and agreed to, then moving it to its own page makes a lot of sense. TonyBallioni (talk) 21:36, 7 July 2017 (UTC)
@TonyBallioni and Scottywong: Correct me if I'm wrong, but I thought the original RfC specified that ACTRIAL would be "3 to 6 months" not "6 months". Kaldari (talk) 21:56, 7 July 2017 (UTC)
@Kaldari: The followup RfC on length closed as 6 months. TonyBallioni (talk) 22:00, 7 July 2017 (UTC)
The consensus was clear enough Ryan: There's a clear consensus for a six-month trial, followed by a one-month period of discussion to determine the trial's effects. Given the wide support and uncontroversial nature of this, combined with me not quite knowing who to ask or where to go to enact this result, I am leaving that to the folks over this way. Ed [talk]|TB| [majestic titan] 1:33 pm, 18 August 2011, Thursday (5 years, 10 months, 22 days ago) (UTC+7). Kudpung กุดผึ้ง (talk) 01:30, 8 July 2017 (UTC)
Some comments on Scottywong's four points:
  1. This doesn't seem too difficult
  2. We should use Confirmed accounts for edit-a-thons etc. I object to putting stuff generated by these initiatives into Draft: space as it doesn't give the sense of astonishment we want to create to hook new editors. The amount of crap created from these account will be limited by the fact that they're somewhat supervised and limited in number. We should also do what we can to have these events focus more on improving existing articles than creating new ones.
  3. As long as the raw data is made public (and WMF has said that it will), I think some of this will take care of itself. We should establish a list of metrics we want to measure and establish normal ranges for these and acceptable deviations and desired improvements. Metrics could include:
    1. Number of new pages created per unit time (same or down)
    2. Number of new pages deleted per unit time (down)
      1. Number of surviving new pages per unit time (same or up)
      2. Percentage of surviving pages per unit time (up)
    3. Number of new AfC submissions per unit time (same or up)
    4. Number of AfC approvals per unit time (same or up)
    5. Number of new registrations per unit time (same)
    6. Number of first edits per unit time (same or up)
    7. Number of first articles per unit time (same or down)
  4. We should be looking at the current trends and level of noise in the metrics I've proposed and determine whether the trial duration would be able to reliably detect the signals we're looking for in this context. One thing is clear, a longer trial is going to give more usable information so we should err towards longer. We can do analysis during the trial and abort if we see strong negative signals. We can start the trial as soon as we have #1 and #2 taken care of and when we beleive we're a month away from being ready to start analyzing results.

~Kvng (talk) 16:44, 8 July 2017 (UTC)

  • My own takes on ScottyWong's points:--
    • 1)This doesn't seem too difficult.Ping Esquivalience who developed something of a proto-type.
    • 2) I find reasonable merit in unbundling the auto-confirmed flag to account-creators.The right-holders are trust(ed)-users and are subject to extremely high levels of scrutiny and the possibilities of a mess-up is considerably less.But that's for another day!As things currently stand; I am against making people send things to draft space.Kvng's suggestion is better and my experience at WP:NPP compels me to believe that the amount of utter-crap that generates out of these projects is marginal.Still, it's best to create a work-group during the trial that will keenly watch these articles.
    • 3)Will be adding later!
    • 4)I will always favor a trial with long span of execution.Anything around 5-6 months is optimum.Whilst the curves of many metric(which we will be consider-ing) will probably display sharp deviations from the current levels; that will hardly be a representative of the long term effects.4-6 months is time optimum enough for a system to stabilise from a sudden change and the graphs thus produced in the later phase will bear the maximum semblance with the effects to be most likely encountered in the long run.Winged Blades Godric 14:02, 9 July 2017 (UTC)
  • I highly encourage making checklists of this kind of stuff. It's essential to keep everyone on the same page of expectations. There is a big difference between "doesn't seem too difficult" and actually getting it done :) —TheDJ (talk • contribs) 09:30, 17 July 2017 (UTC)

Update from Community Tech

Hi everyone: Kaldari, MusikAnimal and I have been talking this week about the possible technical mechanisms for how ACTRIAL could be implemented. I'm going to give a few options, and talk about pros and cons for each; then we can all discuss it and figure things out.

One mechanism that we want to avoid is using the edit filter/abuse filter to prevent non-AC editors from saving their article. Going to the trouble of writing a new page and then not being able to save it is a terrible experience, so we'd like to keep that off the table.

Here are the other options, in order of how much development time it would take to set up:

  1. Use the Titleblacklist and a custom permission error to display a message to the editor. When a non-AC user tries to create a new page, they would see a "Permission error" with a friendly message directing them the Article Wizard (or other options) instead of the editing interface. You can see what this would look like (without the friendly message) by logging out and going to this page. Pros: Very easy to set up. Cons: There's a Visual Editor bug that we'd need the Contributors team to fix, phab:T138715. Also, showing "Permission error" at the top of the page is not super encouraging.
  2. Redirect non-AC users to Article Wizard automatically from the edit page for a nonexistent article. Pros: Medium development time -- probably a week of development, three weeks for security review and deployment -- approximately four weeks total. You don't see "permission error". Cons: Takes people straight to the Article Wizard, without giving other options.
  3. Take the existing work that was done on Extension:ArticleCreationWorkflow, reduce the scope, and deploy a simplified version of it. This would direct non-AC users to a special landing page that presents them with options -- Article Wizard, sandbox, edit something else, etc. Pros: Can offer more options and special messaging. Cons: Longer dev time -- about three weeks of development, three weeks for security review and deployment -- approximately six weeks total.

One question that occurs to us about the messaging: do we explain "4 days, 10 edits", or find a less specific explanation, like "your account isn't old enough to create a new article" or "it doesn't look like you've done enough editing to create an article yet"?

We've submitted a request for funding for the data analyst contractor, but we won't know for a month if the funding's approved -- after August 15th. We could start working on the UI mechanism before we get approval on the research funding, so we don't have to wait until then. I know, August 15th sounds like forever -- we were hoping it would be much faster -- but that's something that's not under our control. -- DannyH (WMF) (talk) 23:13, 7 July 2017 (UTC)

Thanks, DannyH (WMF), my quick thoughts: fine with either options 1 or 3. Strong preference against option 2: sending everyone there automatically would likely just shift the burden to AfC for all the sub-draft quality articles. Option 3 would likely be the best long term option, but the blacklist was probably the preferred route for the "community flips the switch" option, so I don't have opposition to it. The deciding factor for me would be time: how long will it take to fix the bug in option 1 vs option 3, and would there be any considerable timing difference to the actual date of rollout. TonyBallioni (talk) 01:44, 8 July 2017 (UTC)
  1. (edit conflict) All the interface messages were already elaborated by Scottywong and transcuded to the main page WP:ACTRIAL
  2. I don't see why development time is so long. I seem to recall having experimente with this with my own very limited knowledge of web design and php. It wouldn't have taken me weeks.
  3. I think this is almost exactly the solution we are looking for. However, IMO what a new user needs to to see on the registration page already is something like: If you are going to create a new article, we have put in place some exiting features to help you through the process and it will be seen by visitors to Wikipedia as soon as it meets our minimum standards for display. Again, I don't see why development time is so long. I seem to recall having experimented with these messages - and with the Wizard itself - with this with my own very limited knowledge of web design and php. I do think that Jorm's project this is crucial to the future of Wikipedia and as i does not depend on the results of any trials, it's something that could be properly prioritised and begun right now.
Note: In its endeavour to be helpful, even the Wizard itself has become a series of walls of text and is enough to put anyone off from going fufther. It needs revamping by someone who is not a software engineer, but who has majored in Communication studies (or who has preferably done post-grad research in it), and that's a different discipline.
If a workspace for all this is required, I don't see why we can't simply move ACTRIAL and all its associated pages to ACTRIAL 2017, it;s the same project hat is continuing where it was left off.
I would just reiterate again however, that I am of the firm conviction that this should all be a joint development by volunteers and the WMF people, and not another 'top down' project that will be rejected by the community. Also, as regards the timing, we want to avoid another Keyes-Fung phenomenon, and I don't want more emails from the WMF telling me to stay out of it. I have worked for years as hard on this project as anyone else and it's poor form when paid people throw their weight around at the people whose volunteer work pays their salaries - that's a sure fire way to lose the community's cooperation and even more of its trust and respect. Kudpung กุดผึ้ง (talk) 02:06, 8 July 2017 (UTC)
Another option would be to simply archive most of what is at the existing ACTRIAL page (and its associated talk page), and start planning efforts on that page rather than creating a new ACTRIAL2017 page, especially since the trial might extend into 2018. ‑Scottywong| speak _ 18:46, 8 July 2017 (UTC)
Kudpung: Kaldari will be able to answer about development time more intelligently than I will, but here's a start: both Options 2 and 3 involve working with the ArticleCreationWorkflow extension, which is old code that was never actually deployed. The live code that the extension interacts with has changed significantly in the intervening years. This isn't just a matter of throwing some php together -- this is releasing code on one of the biggest and most important websites on the planet. If I've got my own website and I write all my own code, I can do that pretty easily, and if it goes down, then no big deal, I can fix it. Wikipedia is a sprawling, collaborative enterprise, and if you screw something up in the databases, then that changes millions of people's lives for the worse. That's why we need a security review, which can take a couple weeks, and that's why we need to go through the usual deployment channels, which takes a week.
I totally agree with you that this should be a joint development between the volunteers and WMF. That's why we're giving you all the information and options that we know. This won't work if it's top down, and I've been very clear about that with people at the Foundation. Cooperation, trust and respect are essential, and this can only work if each of us is cooperative, trustworthy and respectful. -- DannyH (WMF) (talk) 20:05, 8 July 2017 (UTC)
This research programme largely overlapped many of the items of which the cause and effect will be revealed by ACTRIAL. It also demonstrates that modern statisticians are able to extract almost anything they want from a database and display it in nice, easy-to-read graphs that even a simpleton like me can understand. There were also some people involved in that, namely EpochFail (Aaron Halfaker) and Jonathan Morgan, whose PhDs appear to be concerned human interaction with various stimuli (I guess a bit like my own Communication Studies) who I would like to see working with us on the Landing Page/Article Creation Flow GUIs, and other Wikipedia user interface messages. Is that possible? Kudpung กุดผึ้ง (talk) 23:44, 8 July 2017 (UTC)
Aaron and Jonathan are consulting with us on this project; they were part of the group that decided to help run the ACTRIAL experiment, and Aaron recommended the analyst that we're hoping to contract with. -- DannyH (WMF) (talk) 03:40, 9 July 2017 (UTC)
  • DannyH (WMF), with respect to my post above, you said: ...we won't know for a month if the funding's approved -- after August 15th. It's therefore not a foregone conclusion that the funds will be approved and another five weeks to wait is really stretching it a bit. What's the contingency plan if they're not? Kudpung กุดผึ้ง (talk) 23:52, 8 July 2017 (UTC)
The contingency plan is to try again, or figure something else out. We were disappointed and frustrated yesterday when we found out that we have to wait that long. But I don't want to make any promises that we can't keep, so I have to be honest about where we are. -- DannyH (WMF) (talk) 03:45, 9 July 2017 (UTC)
No objections, but as that page contains all the essential elements, keep the sections: Draft messages for non-autoconfirmed users and Statistics (the lists). This save valuable time by helping to reduce a lot of the work of the WMF to the physical coding of anything that is required, and providing the 'before & after' graphs. Let's also not forget user:Scottywong/Article creation stats which contains important models for graphs which the WMF found so hard to replicate.
However, I do implore all concerned to finally read that page rather than just skim over it. I note that many of the questions the WMF and newcomers to ACTRIAL are still posing clearly demonstrate that they have not read it, or have not read it properly. Kudpung กุดผึ้ง (talk) 01:46, 9 July 2017 (UTC)
No objections, thanks. ‑Scottywong| confer _ 17:48, 11 July 2017 (UTC)
  • At this rate, all that I want is for ACTRIAL to be implemented before the quality control problem swells to some figure like 25,000 unreviewed pages – 0.5 percent of all articles. The more readers that see the abysmal quality of new articles, the more likely readers will become frustrated and use other resources instead. Esquivalience (talk) 03:06, 9 July 2017 (UTC)
  • I agree with Esquivalience: in its early days Wikipedia was derided for being unreliable due to the "anyone can edit" aspect. In the last few years, I have seen students and professors alike refer to it in much higher esteem. I fear a reversal of this trend may occur if we do not implement ACTRIAL soon. I suggest using the blacklist method as soon as possible to start ACTRIAL as specified above. DrStrauss talk 16:48, 9 July 2017 (UTC)
  • I also agree with Kudpung, Esquivalience and DrStrauss - a tincture of common sense and experience at AfC and NPP are telling ingredients. I'm pinging Lourdes because, if I'm not mistaken, he writes code and I hope he's been following this discussion. Atsme📞📧 19:36, 9 July 2017 (UTC)
    • As I replied to DrStrauss on my talk page, the huge advantage of working with the WMF now to implement the 2011 consensus is that we don't need another RfC to move forward. They also have the resources to help with things that volunteers could do but it would be much easier to have a paid person do it: write the code, collect the stats, etc. These should be done with community collaboration, but working with the WMF on this is vastly preferable to flipping the switch ourselves without them in my opinion. I do think that is an option if this all falls through, but working with the WMF is preferable as Scottywong points out above even if we don't need to. We can start working on the project now: thinking through and drafting information to inform affected groups, working on the article wizard, and the other tasks Kudpung has pointed out at WT:ACTRIAL. I will have more time later tonight to point out some other tasks that I think need to be done, but there is in fact a lot of work that can be done before the switch is flipped. We don't need to wait until they hire an analyst to do it. TonyBallioni (talk) 19:46, 9 July 2017 (UTC)
      • It's looking like we're almost at the stage where the switch is about to flip with both WMF and community consent. My question is how will this happen? From the discussion, it looks like the preferred method is the blacklist, so which admin will implement it, and when? What tasks have yet to be completed by both the community and the WMF which are required to activate ACTRIAL? DrStrauss talk 20:12, 9 July 2017 (UTC)
The comment by TonyBallioni above is so precise it obviates the need for me to repeat it. At the moment we're just going to have to be patient. I've been waiting (and pleading) for this to happen for nearly 6 years so although it ought to be done quickly, it won't break Wikipedia to wait a couple more weeks if we have to. That said, the WMF team is waiting for funds for the stats, but that should not hinder the required software developments in the meantime.
The various tasks involved are already outlined at WT:ACTRIAL - it's more than just flipping a switch because circumstances (and code) have changed since we drew up the original ACTRIAL project.Kudpung กุดผึ้ง (talk) 20:55, 9 July 2017 (UTC)
  • But to answer the original question. I'd go with option 3. The current article creation interface is a tragedy and it needs some serious work. Esquivalience (talk) 01:26, 10 July 2017 (UTC)
    • It sounds like there's still some debate about whether option 1 or option 3 is the best implementation. Unfortunately, I can't say for certain which would be quicker to implement (option 1 is probably faster but depends on a different team to fix the bug). Either way, we can probably finish either of them before we have our data analyst (since that likely won't happen until late August). Personally, I would love to see option 3 implemented, as I think it would be the best experience for new users, but I support whatever decision you guys settle on. Kaldari (talk) 18:31, 10 July 2017 (UTC)
      • Kaldari, if that is the case I would say go with option 3. I think the mood here is "do this right". I think if both would happen before the data analyst gets on, its worth an extra week or two, especially if we would only have to work with one team of WMF developers. TonyBallioni (talk) 19:48, 10 July 2017 (UTC)
      • I would also prefer that we get it right (with option 3) rather than try to implement something that will still need to be redone and doesn't really save all that much time. It doesn't seem that there is a viable solution that would work in the very short term. Mduvekot (talk) 20:59, 10 July 2017 (UTC)
I concur with that. What we need to do is get those mock ups and flow chart of Jorm's over here so we have a basis to work from and to see if it is/was supposed to be in addition to the wizard r a replacement for it. In any case, users need to be given some hint of what they can and can't do right at the moment of registration. Kudpung กุดผึ้ง (talk) 01:21, 11 July 2017 (UTC)
I couldn't agree more with the preceding sentence. What such hints need to convey – not of course in those words – is something like this: Noyster (talk), 08:36, 11 July 2017 (UTC)
There are ways of implying the same thing but in a positive/encouraging tone and using the KISS principle: If you are going to create a new article, we have put in place some exiting features to help you through the process and it will be seen by visitors to Wikipedia as soon as it meets our minimum standards for display. Kudpung กุดผึ้ง (talk) 09:46, 11 July 2017 (UTC)

I agree with your comment on tone. While spammers, paid editors and autobiography writers are a large part of the problem at NPP and AFC, we must ensure that we don't drive away potential long-term, good-faith Wikipedians. As has been mentioned above, many of the help pages will need updating for consistency with the new policy, I suggest a transcluding template that explains ACTRIAL. Any thoughts on that method? DrStrauss talk 12:55, 13 July 2017 (UTC)

ACTRIAL is a trial. I think it probably unwise to start calling it a policy or even a policy change. That's why we're going to tun the trial. The results of the trial could go either way, but that should not be used as an excuse for the WMF to delay urgently needed software already. Kudpung กุดผึ้ง (talk) 01:59, 16 July 2017 (UTC)

Hi everyone, we have some progress to report on the Community Tech side. We're able to work with the analyst a lot sooner than we thought we would, and Kaldari and I had an initial meeting with him on Friday. The analyst is Morten Warncke-Wang, aka User:Nettrom -- he's a long-time Wikipedian who's been running SuggestBot since 2010. He's a PhD who's published research on the production of quality content on Wikipedia, and he's done a lot of work with Aaron Halfaker to develop the article quality model that's used in ORES. He's really excited about working on this project -- it's not often that you get the chance to make a noticeable change in a core workflow on English Wikipedia. :)

Morten's starting to build a Research page on Meta -- Research:ACTRIAL -- and the first part is figuring out the research questions. He's using the list of questions on the ACTRIAL page as a guide, and also thinking about other questions that weren't technically feasible when that list of questions was compiled in 2011. The plan is for him to come up with the first draft of that list this week, so that we can all discuss it and work on a final list together.

On the tech side, we started working last week on a simple version of ArticleCreationWorkflow that will redirect non-autoconfirmed users who try to create pages to a new special page. That's in progress right now -- it's phab:T170354 if you want to follow it. So that'll be ready soon, and we should all talk about what's going to happen from that page. If you're familiar with the Phabricator ticket system, and you want to track all of the related tickets, the main ticket is phab:T170851. Right now, there's only the one ticket, but there will be more as we go along.

So now that we're getting into the planning, where should we have these conversations? Should we be moving to the main ACTRIAL talk page? Also, I saw the conversation on Jimbo's talk page -- should we be advertising this more widely? -- DannyH (WMF) (talk) 18:27, 17 July 2017 (UTC)

Yeah, let's move the planning and discussion around ACTRIAL to Wikipedia:Autoconfirmed article creation trial and the talk page there. It looks there are already some discussions underway there. Kaldari (talk) 18:39, 17 July 2017 (UTC)
Agreed that ACTRIAL/ACTRIAL talk is the correct place to hold it now. @DannyH (WMF) and Kaldari: I think the post at Jimbo talk and VPP has served the objective of making it more well known to people, and showed that while there are still a few people who oppose, the community is largely behind this still and there really isn't demand for an RfC. In terms of letting more people know: we've directed people to the pages now and invited comments. I also tend to agree with Kudpung that these things are easier to do with a work group and then make the more specific plan widely known (of course the work group being on-wiki so anyone who wants can take part). TonyBallioni (talk) 19:14, 17 July 2017 (UTC)

(Just flagging that T138715 is now marked as resolved. Best, Elitre (WMF) (talk) 14:57, 19 July 2017 (UTC))

Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:The_future_of_NPP_and_AfC/Archive_4&oldid=1074880753"