Wikipedia:Bots/Requests for approval/JJMC89 bot III

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was

Approved.

JJMC89 bot III

Operator: JJMC89 (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 07:23, Sunday, February 24, 2019 (UTC)

Function overview: Process WP:CFD/W and its subpages excluding WP:CFD/W/M

Automatic, Supervised, or Manual: Automatic

Programming language(s): Python

Source code available: ~~cfd.py~~ cfdw.py

Links to relevant discussions (where appropriate): WP:BOTREQ#Categories for Discussion bot (permalink)

Edit period(s): Hourly

Estimated number of pages affected: Millions

Namespace(s): Many

Exclusion compliant: delete/move: no; edit: yes

Adminbot: Yes

Function details: Process WP:CFD/W and its subpages excluding WP:CFD/W/M, moving and deleting categories and re-categorizing pages as specified

Notes

This task takes over work being done by Cydebot (BRFA · contribs · actions log · block log · flag log · user rights) (Task: 4), operated by Cyde (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search).
Notices: WP:AN, WT:CFD, User talk:Cyde

Discussion

Since Cyde is inactive, I am requesting t:o take over the task so that bugs can be fixed and feature requests implemented. Additionally, Cydebot will stop functioning at the end of March due to the Toolforge Trusty deprecation unless it is migrated to Stretch. The code is the based on the code that Cydebot is running. — JJMC89 (T·C) 07:23, 24 February 2019 (UTC)[reply]

You say it's "based on" Cydebot's code, presumably meaning that you made changes. Can you please summarize the effects of these changes? עוד מישהו Od Mishehu 08:45, 24 February 2019 (UTC)[reply]

Cyde's code is part of Pywikibot, but Cydebot is not using the current version. I haven't made changes to the scripts yet, but other maintainers have. There haven't been any changes that would change the functionality of this task. — JJMC89 (T·C) 22:07, 24 February 2019 (UTC)[reply]

I've now rewritten the code that parses the working page. The code that moves/deletes the categories and re-categorizes the pages is still the same as Cydebot's. — JJMC89 (T·C) 07:32, 15 March 2019 (UTC)[reply]

Fluxbot is already approved for this task. {{3x|p}}ery (talk) 20:00, 24 February 2019 (UTC)[reply]

@Pppery: As far as I can tell the last edit Fluxbot made under task1 (cfds) was in July 2017. Pinging @Xaosflux who may be able to fill us in --DannyS712 (talk) 20:06, 24 February 2019 (UTC)[reply]

As an AWB bot, Fluxbot doesn't operate without operator intervention. It also isn't an adminbot. — JJMC89 (T·C) 22:07, 24 February 2019 (UTC)[reply]

@JJMC89, Pppery, and DannyS712: yea, Fluxbot for CFDS is only on-demand, I used to process CFDS regularly, but more efficient bots came around. — xaosflux ^Talk 22:27, 24 February 2019 (UTC)[reply]

Thanks, @JJMC89.

In principle, I very much welcome idea of a new bot with extended functionality. (There are some major gaps in the current feature set).. However, I think that @Black Falcon was a bit hasty in posting the request at WP:BOTREQ when the discussion at WT:Categories_for_discussion/Working#Cydebot_replacement had few participants (I think only 4) and had not been widely notified; the suggested extra functionality needs more discussion.

However, it now turns out that a replacement is needed soon, so thanks to JJMC89 for stepping up in the nick of time.

So I hope that any new bot will run initially with the same functionality as CydeBot. Any enhancements need a clear consensus, which we don't yet have. --BrownHairedGirl (talk) • (contribs) 05:05, 10 March 2019 (UTC)[reply]

At a minimum, the new bot should process the main /Working page:

Deleting, merging, and renaming (i.e. moving) categories, as specified, with appropriate edit summaries.

Deleting the old category with an appropriate deletion summary.

In the case of renaming, removing the CfD notice from the renamed category.

Ideally, it would also do some or all of the following:

Process the /Large and /Retain subpages.

Accept manual input when a category redirect should be created—for example, by recognizing leading text when a redirect is wanted, such as * REDIRECT Category:Foo to Category:Bar.

Recognize and update category code in transcluded templates. This would need to be discussed/tested to minimize errors and false positives.

Recognize and update incoming links to the old category. This would need to be discussed/tested to minimize errors and false positives.

-- Black Falcon ^(talk) 20:48, 18 February 2019 (UTC), Wikipedia:Bot requests#Categories for Discussion bot

@BrownHairedGirl: Thanks for pinging me, and I don't disagree with you, fundamentally. More participation would undoubtedly have been better, and I probably should not have assumed many people watchlisted WP:CFD/W, but all the manual work required to close out Cydebot's processing was becoming quite tiresome after two months. In terms of the functionality I requested, only the last three items are enhancements compared to what Cydebot currently does or did when it was functioning properly, and I did note the last two needed more discussion/testing. As I mentioned at WP:BOTREQ, I would be happy if a bot did just the first three items properly. -- Black Falcon ^(talk) 06:06, 10 March 2019 (UTC)[reply]

Thanks, @Black Falcon.

I haven't so far seen anything on the list of suggested improvements, which I would dislike, and I see much I am v keen on.

In particular, the work required to close out CydeBot's efforts is v onerous, and I dearly wish that could be improved. I have been cleaning up after WP:CFD 2019 February 16#North_Macedonia, which renamed ~650 categories, and so far it has taken me about 8 hours to process about 400 of those. There has to be a better way somehow.

I agree that the redirects thing should be simple. However, the precise details of what a bot would do in next two cases needs a lot of scrutiny. It would be easy for an ill-specified bot to wreak much havoc in templates. --BrownHairedGirl (talk) • (contribs) 06:24, 10 March 2019 (UTC)[reply]

This BRFA is to take over Cydebot's current functionality. The enhancements, particularly the last two points, are out of scope. After this is up and running (with any bugs are worked out), I'll be happy to look at making enhancements that have consensus. — JJMC89 (T·C) 18:36, 10 March 2019 (UTC)[reply]

Thanks, @JJMC89. Sounds great. --BrownHairedGirl (talk) • (contribs) 18:38, 10 March 2019 (UTC)[reply]

Understood, and thank you. -- Black Falcon ^(talk) 20:19, 10 March 2019 (UTC)[reply]

{{BAG assistance needed}} I would like to get this up and running before Cydebot stops functioning. — JJMC89 (T·C) 07:32, 15 March 2019 (UTC)[reply]

Note: Cyde has migrated Cydebot to Stretch, so it is no longer in danger of dying at the end of the month. — JJMC89 (T·C) 00:54, 21 March 2019 (UTC)[reply]

@JJMC89: I suspect there could be issues with both of these bots trying to run the same page on top of each other - do you want to proceed as a backup plan, or do you want to try to work in tandem going forward? — xaosflux ^Talk 19:40, 21 March 2019 (UTC)[reply]

@Xaosflux: I'm not sure, but there could be. It would probably be best to disable Cydebot since Cyde isn't fixing any of the bugs, which is the original reason it was requested for someone else to take over the task. — JJMC89 (T·C) 03:28, 22 March 2019 (UTC)[reply]

Comment. Cydebot stopped running more than 24h ago, and we are currently paralyzed with categories. It were great if this request could be approved.--Ymblanter (talk) 19:04, 23 March 2019 (UTC)[reply]

I agree with @Ymblanter. Please can we make the switchover ASAP? --BrownHairedGirl (talk) • (contribs) 19:42, 23 March 2019 (UTC)[reply]

Let me reactivate {{BAG assistance needed}} Galobtter (pingó mió) 19:55, 23 March 2019 (UTC)[reply]

@Galobtter: this request is already 'active' - the task is not, as it hasn't even trialed, yet. — xaosflux ^Talk 20:10, 23 March 2019 (UTC)[reply]

Xaosflux The "reactivating" was of the BAG assistance needed template, to get a BAG to come by and issue a trial (since Cydebot is crucial to WP:CFD and so it'd be good for this to be trialed and approved soon as mentioned above). Galobtter (pingó mió) 20:15, 23 March 2019 (UTC)[reply]

@JJMC89: do you have "human friendly" documentation (e.g. what you will do, when you will do it, and under what conditions) for this? Since this bot is going to almost entirely be edits and actions you are responsible for, but will be triggered by other admins - it should be clear to other admins what their requests will do. — xaosflux ^Talk 20:43, 23 March 2019 (UTC)[reply]

@Xaosflux: If the information at WP:CFD/AI and WP:CFD/W, including the HTML comment, aren't sufficient I can write something more detailed. The intent is to function as Cydebot is intended to function. — JJMC89 (T·C) 21:21, 23 March 2019 (UTC)[reply]

@BrownHairedGirl: did you have concerns that getting started under those "rules" would be an issues (would it be worse then just not running at all?) — xaosflux ^Talk 21:41, 23 March 2019 (UTC)[reply]

(ec) @Xaosflux: Cydebot has done its job almost unchanged for ~13 years. AFAIK, the only significant change was when category pages became movable in 2014. So the bot's functions are stable and well understood by the admins who instruct it. Since JJMC89's bot is designed to be a drop-in-replacement, I don't see any need for more docs as a precondition -- provided the new bot passes its test, without which this discussion is moot anyway.

Of course, if more functions are added, then docs will be appropriate. But right now a backlog is building up, so we need a bot and need it urgently. --BrownHairedGirl (talk) • (contribs) 21:47, 23 March 2019 (UTC)[reply]

Approved for trial (250 edits). Please provide a link to the relevant contributions and/or diffs when the trial is complete. (edits/action) OK, do a short trial for proof of concept. — xaosflux ^Talk 23:05, 23 March 2019 (UTC)[reply]

Trial complete. The trial was run on this version of WP:CFD/W. edits (deleted) logs Since the bot isn't an admin yet, categories were tagged for deletion instead of deleted. Some of them got tagged multiple times, but this is just an artifact of not being an admin and me stopping and starting the script to limit the trial edits. It has the same problem that Cydebot does where it doesn't delete (tag for deletion) the original category like it should. This should happen right after it finishes emptying the category. I think the issue is either caching or MediaWiki thinks the category still has members. (It checks that the category is empty before trying to delete.) You'll see for the categories being emptied, if the script is run again it will delete (tag for deletion) the second time around. This doesn't happen for moves/merges since there is a check that the destination category does not exist at the start of processing. (It will always exist a second time around.) This is something I can work on fixing. The fix could be 1) adding a delay before checking that the original category is empty, 2) moving categories without leaving a redirect, or 3) removing the check that the destination category must exist. — JJMC89 (T·C) 01:55, 24 March 2019 (UTC)[reply]

Awesome! I have two thoughts, and hopefully they are both quick fixes. First, the summary should link only to the daily log page, not the section—partly, this would be to keep the summary shorter (which will be important when dealing with long category names), and partly because the section name is not always the same as the category name (e.g. group nominations). Second, and more importantly, moving categories should be done without leaving a redirect. Another bot automatically recategorizes any pages placed in a redirected category, and so auto-creating category redirects dramatically increases the likelihood for miscategorization—see WT:CFD/W#Cydebot replacement for just a few examples. This will take care of the issue in cases of renaming but not merging, where solutions 1 or 3 may be needed. -- Black Falcon ^(talk) 03:01, 24 March 2019 (UTC)[reply]

@Black Falcon: Changing the edit summary is easy. Personally, I dislike having to find the correct section when a daily log is linked without a section. The code doesn't assume that the section is the name of the category. It looks for the category as a heading or a link to the category in nomination part of each discussion on the daily log page and links to that section. I don't think there is a danger of running out of space in the edit summary, but if it is the appearance, what about piping it like [[Wikipedia:Categories for discussion/log/<date>#Section|Wikipedia:Categories for discussion/log/<date>]]? In not, I'll just use the link directly from CFDW. Something like this should suppress redirects when moving. Since we're moving without leaving a redirect, I could just delete the category regardless of it it has members for merge/empty. Otherwise, I will have to do some live testing figure out a good delay before checking if it is empty. — JJMC89 (T·C) 04:56, 24 March 2019 (UTC)[reply]

@JJMC89: Many thanks for running the trial. That looks good.

I presume that the inclusion of section links in the edit summaries is an artefact of me having left the section links in the WP:CFDW pages: here[1] and [2].

I did so because I had noticed that CydeBot just used whatever was in that header, and it seemed to me that there is no advantage in omitting them now that the length of edit summaries is almost unconstrained. However, I had not foreseen that it would coincide with the bot trial, so sorry for any confusion.

I don't see any benefit in piping the summaries to hide the section link. That just wastes space in the summary and misleads the readers.

As to redirects, there are many situations where it is appropriate to have them, and it is much easier (and less error-prone) to delete the un-needed redirects than to create them when omitted. Personally, I would favour a flexible approach with some new language of bot instruction. So instead of an abrupt change from always-on to always-off, we need some discussion on how best to handle them.

The big picture is that on both edit summaries and redirects, JJMC89's bot seems to be behaving exactly as CydeBot does. @Black Falcon, I thought we had had agreed to leave the possibility of changes to the bot's behaviour for a separate discussion? I agree that we need some changes, but I don't agree that the issues in either case are quite as you describe them, and I am disappointed to see them being raised here and now. They both need more discussion at WT:CFDW, with more people involved.

So please can we keep this process focused on the v urgent task of getting a drop-in replacement for the now apparently-defunct Cydebot? That was the basis on which Xaosflux authorised this trial, so I am surprised to see proposed changes.--BrownHairedGirl (talk) • (contribs) 08:02, 24 March 2019 (UTC)[reply]

Section links do get carried over from CFDW; however, I did add a feature to try to add section links when not provided because I find them helpful. (See some details in my reply to Black Falcon above.) Based on Cydebot's code, Cydebot is supposed to be deleting the redirects, but it isn't. My bot inherited that issue since it uses the same code for that. What we do with the redirects right now isn't a big concern for me. A granular system can be worked out later. For now, I can either keep doing what Cydebot has been doing or fix the code to delete the redirects. — JJMC89 (T·C) 08:35, 24 March 2019 (UTC)[reply]

I also do not think redirects should be indiscriminately deleted. We are checking the backlinks anyway, and deleting a redirect takes much less time than checking backlinks anyway.--Ymblanter (talk) 09:39, 24 March 2019 (UTC)[reply]

Please can we just stick for now with what Cydebot was actually doing? --BrownHairedGirl (talk) • (contribs) 10:21, 24 March 2019 (UTC)[reply]

This is exactly what my remark was about.--Ymblanter (talk) 10:35, 24 March 2019 (UTC)[reply]

Sorry, @Ymblanter. Indeed it was, and I meant to acknowledge that but didn't. (I'm a bit sleepy today). --BrownHairedGirl (talk) • (contribs) 11:25, 24 March 2019 (UTC)[reply]

Done Now back to matching Cydebot's code for redirects. — JJMC89 (T·C) 16:41, 24 March 2019 (UTC)[reply]

@BrownHairedGirl: As far as I can tell, auto-creating tens of thousands of cat-redirects was never an approved function for Cydebot, and a bot should not be mass-creating pages when there is such a high error rate. I agree that there some situations where a cat-redirect is appropriate but strongly disagree that it is much easier (and less error-prone) to delete the un-needed redirects than to create them when omitted. It is, in my experience (having spent countless hours deleting inappropriate cat-redirects), more time-consuming to properly delete a page than to undelete it or just type {{Category redirect|Foo}}, and the risk of error is inherently greater when an action is necessary to avoid errors (i.e. deleting an inappropriate cat-redirect) than when an action is optionally helpful (i.e. creating an appropriate cat-redirect).

@Ymblanter: Re: checking backlinks, unforrtunately, I think that will continue to have to be performed manually regardless of whether a cat-redirect is retained. A bot may be able to help, but I don't see a bot being able to account for and fix all possible scenarios of incoming links. -- Black Falcon ^(talk) 17:37, 24 March 2019 (UTC)[reply]

@JJMC89: I'm sorry, my intention was not to create more work for you. I will be appreciative of what you're doing however the code ends up, but I struggle with including a function that was never approved to begin with. In general, I've always struggled with the idea of bots auto-creating pages without narrow parameters and/or human input. As you pointed out, Cydebot was not supposed to be leaving redirects but started to do so at some point, and for very good reason in my view. -- Black Falcon ^(talk) 17:37, 24 March 2019 (UTC)[reply]

@Black Falcon, Leaving redirects is a normal function of page moves. It is appropriate in some cases, inappropriate in others, with lots of grey in between. I personally think you are being way too absolutist about this, and that there are better solutions than simply disabling all creation of redirects. That's why any change needs to be discussed.

But this page is not the place to be having that discussion.

And the effect of your insistence on using this discussion to change the behaviour of the bot is to delay the approval of the new bot at a point when the old bot is apparently gone.

Please can you explain why you object to having a proper consensus-forming discussion on this, e.g. at WT:CFDW? And why instead you are trying to use BRFA to unilaterally force a change in the way that the bot had worked for nearly 5 years? This is not the way that consensus is built.

} --BrownHairedGirl (talk) • (contribs) 18:07, 24 March 2019 (UTC)[reply]

Categories are not the same as other pages when it comes to leaving redirects after moves, because bots patrol cat-redirects and move any contents into the target category. When inappropriate cat-redirects are created, this leads to pages being miscategorized.

Please don't misunderstand me, I do agree that there is "lots of grey in between". Between the two extremes of clearly appropriate cat-redirects (e.g. hyphen-to-dash redirects) and clearly inappropriate cat-redirects (acronym-to-full name redirects), there are many others that are harder to categorize and even more that are better judged based on usefulness than appropriateness. This is precisely why, in my view, a bot should not be mass-creating cat-redirects without human intervention.

I do not object to a consensus-forming discussion, and I am agree we should probably continue this discussion elsewhere. What I do object to is purposely retaining a bug that performs a function was never approved in the first place. When it comes to auto-creation of pages, consensus historically has been to take the more cautious approach that minimizes the likelihood of errors. And from a BRFA standpoint, it is simply not feasible to demonstrate that the bug results in an acceptably low error rate. -- Black Falcon ^(talk) 18:47, 24 March 2019 (UTC)[reply]

@Black Falcon, when the bot has been doing that for five years, it's a WP:IMPLICITCONSENSUS. A discussion may produce a different result, but it does need a discussion.

Having the bot not create redirects would also create a high error rate. I am very well aware of what RussBot does, and why ambiguous redirects should not be retained, but simply switching from one binary option to another creates as many problems as it solves. Describing the current process as a bug with a simple fix misrepresents a complex reality.

so ... please can you clarify whether you oppose or support JJMC89's bot going live to do exactly as Cydebot did? --BrownHairedGirl (talk) • (contribs) 20:16, 24 March 2019 (UTC)[reply]

I do not think there is a "simple fix", but I think not creating redirects is a better fix and disagree it would "create as many problems as it solves". I acknowledge that this will result in some appropriate redirects not being created, which inconveniences navigation. However, the alternative results in inappropriate redirects being created, which produces actual errors through miscategorization. I would rather take inconvenience over outright errors. And, yes, we can consider other (non-binary) options, but those have not been tried before and would need to be discussed and tested.

Regarding consensus... a lack of any alternative is not indicative of consensus—implicit or otherwise. Quite simply, a malfunctioning bot was better than none at all. So, in response to your ultimate question, I will not oppose JJMC89's bot going live no matter what. My preference remains for the bot to do what Cydebot was approved to do, and not to continue mass page-creation on the basis of inertia stemming from an unfixed bug, but ultimately I will defer to BAG on the appropriate course of action. -- Black Falcon ^(talk) 02:51, 25 March 2019 (UTC)[reply]

It's bit tedious that @Black Falcon persists in describing as a "bug" something for which there is no consensus on optimal behaviour. I have seen no evidence that CydeBot was approved either to create redirects or not to create redirects:WP:Bots/Requests for approval/Cydebot_4 does not mention the possibility of redirects, either in the proposal or in the discussion. CydeBot was approved long before category page became movable, and its actions since moving was enabled have conformed to the default process for moving pages, i.e. leave a redirect.

This is simply an issue which has not been resolved since category pages became movable in 2014. It needs a consensus, but consensus-building is not helped by one editor trying to sway BAG by describing the bot's failure to conform to their personal preference as a "bug". --BrownHairedGirl (talk) • (contribs) 15:31, 25 March 2019 (UTC)[reply]

{{BAG assistance needed}} — JJMC89 (T·C) 04:27, 3 April 2019 (UTC)[reply]

Approved. This task shall proceed as described to keep things flowing. As far as the issue with the redirects is concerned, the "implicit consensus" will stand for now until such time as a discussion deems otherwise. Primefac (talk) 21:47, 27 April 2019 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.