User:Enterprisey/AIV analysis

I analyzed some AIV reports for the "Responder" proposal (permalink), to see what the usual AIV response time was and to see how many edits vandals make between when they get reported and when they get blocked.

TODO

verify I didn't get timezones wrong on the "reports per hour of day" and "response time vs hour of day" graphs
graph "how many AIV reports have been responded to" vs "hours after report made" - and once that's done, maybe require the report to sit for X minutes in the main proposal?
Jan-Jun 2022 (inclusive)

Background

Patrollers' time is wasted when they have to keep reverting the same vandal instead of checking new edits. We could fix that by letting them block vandals for an hour while they wait for an admin to take action. A process for that would use up a lot of time and effort, so we'd better check if it's worth it.

Definitions

A case is an AIV report plus a resolution (blocked, report expired, etc), perhaps with some edits made while the report was active.

Data

If you're interested, please help out with analyzing the data! I did a pretty amateurish job and I'm sure we could get more information out of this.

One year

The year from 2022-09-01 to 2023-09-01 was analyzed.

There were 37 287 cases overall. 45 997 edits were made while reports were waiting, of which 36 096 were both reverted and made by a user who got blocked.

Looking at cases that ended in a block ("block" cases): in about 3987 of those cases, edits occurred at the rate of greater than 1 edit per minute. Those cases resulted in 28695 edits. Such cases were about 60% of "block" cases with 2 or more edits and about 12% of all "block" cases.

Worst cases (first link label is number of edits while report active, second link label is response time; all of these ended in a block):
Reverted edits: 417 (8m), 132 (2m), 113 (35m), 95 (2m), 87 (19m), 83 (7m), 83 (1m), 79 (7m), 78 (37m), 74 (3h), 67 (59m), 66 (22m), 63 (14m), 59 (16m), 58 (32m), 57 (38m), 56 (3m), 55 (1h), 54 (5h), 54 (13m), 54 (42m), 54 (3h), 53 (35m), 52 (3m), 51 (0m), 51 (3h), 51 (3m),
All edits: 424 (8m), 188 (2m), 132 (2m), 123 (2m), 113 (35m), 98 (35m), 94 (3h), 94 (7m), 91 (5h), 88 (7m), 87 (19m), 86 (4h), 83 (1m), 82 (26m), 81 (1m), 78 (37m), 76 (3m), 67 (59m), 66 (22m), 64 (14m), 63 (59m), 62 (16m), 60 (42m), 58 (32m), 57 (38m), 57 (3h), 56 (2h), 55 (1h), 55 (4h), 54 (13m), 54 (5h), 54 (3h), 52 (3m), 51 (0m), 51 (3m), 50 (1h)

Rapid edits:

Old data

Very brief summary: Vandals made between about 11 210 and 15376 reverted edits while a report at AIV was active during October through December 2020.

Description of the data format: https://git.sr.ht/~enterprisey/aiv-analysis-jan-2021/tree/master/item/case_format.txt
October 2020: https://tools-static.wmflabs.org/apersonbot/aiv-analysis-oct-2020.v1.json
November 2020: https://tools-static.wmflabs.org/apersonbot/aiv-analysis-nov-2020.v1.json
December 2020: https://tools-static.wmflabs.org/apersonbot/aiv-analysis-dec-2020.v2.json
Fall 2021: https://tools-static.wmflabs.org/apersonbot/2021-09-01T00%3A00%3A00Z--2021-12-01T00%3A00%3A00Z--cases.0.json

Notes on data accuracy:

I couldn't parse some of the reports (21 each for November and December 2020), so the numbers are slightly too low.
Edits without the "reverted" tag still could've been reverted, just not directly. The true number of reverted edits is therefore between the official number and the total (reverted + non-reverted) number.
Editors who I couldn't find a block for still could've been blocked, perhaps via a larger range block. More editors should thus have been counted as "blocked".

All months

October + November + December 2020: there were 10209 reports total. (14:00 UTC is 9:00 AM Eastern U.S., by the way)

Note that the number of reverted edits by hour is relatively stable, meaning that adding more admins at night won't fix this. There are lots of bad edits getting made even when there are lots of admins around (that is, when the response time is the lowest, at 14 UTC).

October 2020

There were 3777 (parseable) reports in total. The targets of those reports made 5991 edits while the reports were active, of which 4997 were reverted and 994 weren't. 3329 of the reports ended in a block; the targets of those reports made 4937 edits, of which 4533 were reverted and 404 weren't.

2193 reports were of IPs, and 1584 were of accounts. For just reports ending in a block, 1947 were of IPs and 1382 were of accounts.

(not shown: 2074 cases with no revs or 593 cases with 1)

November 2020

There were 2954 (parseable) reports in total. The targets of those reports made 4638 edits while the reports were active, of which 3574 were reverted and 1064 weren't. 2525 of the reports ended in a block; the targets of those reports made 3766 edits, of which 3241 were reverted and 525 weren't.

1597 reports were of IPs, and 1357 were of accounts. For just reports ending in a block, 1358 were of IPs and 1167 were of accounts.

December 2020

There were 3478 (parseable) reports in total. The targets of those reports made 4747 edits while the reports were active, of which 3886 were reverted and 861 weren't. 2934 of the reports ended in a block; the targets of those reports made 3798 edits, of which 3436 were reverted and 362 weren't.

2011 reports were of IPs, and 1467 were of accounts. For just reports ending in a block, 1710 were of IPs and 1224 were of accounts.

Scatter plot (00:00 on 1 December 2020 to 16:27 on 6 December 2020):

Other conclusions

In about 1043 cases, edits occurred at the rate of greater than 1 edit per minute. That's about 59% of all cases with 2 or more edits and about 10% of all cases. Those cases resulted in 8956 edits.

Worst cases during Oct/Nov/Dec 2020 (first link label is number of edits while report active, second link label is response time; all of these ended in a block):
Reverted edits: 102 (1h), 95 (19m), 93 (27m), 76 (2h), 69 (1h), 63 (22m), 61 (44m), 57 (1h), 56 (31m), 56 (3h), 55 (38m), 55 (21m), 54 (1h), 53 (1h), 53 (9m), 50 (6h), 50 (3h), 49 (1h), 49 (12m), 44 (19m), 42 (31h), 42 (17m), 39 (3h), 39 (2h), 37 (3h), 37 (6m), 37 (31m), 36 (6m), 35 (2h), 35 (9m), 35 (15m), 35 (1h), 35 (21m), 32 (14m), 32 (26m), 32 (15m), 32 (1h), 32 (1h), 32 (50m), 32 (2h), 31 (2h), 30 (6m), 29 (1h), 29 (2h), 29 (1h), 28 (1h), 28 (26m), 27 (57m), 27 (35m), 27 (13m), 27 (10m), 26 (9m), 26 (13m), 26 (24m), 26 (34m), 26 (11m), 26 (9m), 25 (12m),
All edits: 220 (31h), 105 (1h), 95 (19m), 93 (27m), 77 (2h), 70 (1h), 68 (4h), 67 (38m), 65 (3h), 64 (22m), 63 (21m), 61 (44m), 59 (2h), 57 (1h), 57 (31m), 56 (3h), 55 (1h), 55 (3h), 54 (9m), 54 (1h), 50 (6h), 49 (12m), 49 (1h), 49 (4m), 45 (2h), 44 (19m), 43 (31m), 42 (17m), 41 (21m), 39 (3h), 38 (14m), 37 (11h), 37 (6m), 37 (3h), 36 (6m), 36 (2h), 36 (15m), 35 (9m), 35 (2h), 35 (1h), 33 (15m), 33 (1h), 33 (50m), 32 (26m), 32 (2h), 32 (1h), 32 (1h), 30 (6m), 30 (1h), 30 (4h), 29 (2h), 28 (26m), 28 (10m), 28 (13m), 28 (4h), 28 (39m), 28 (1h), 27 (35m), 27 (57m), 27 (6h), 27 (13m), 27 (34m), 26 (1h), 26 (11m), 26 (9m), 26 (24m), 26 (9m), 25 (13m), 25 (12m)

Fall 2021

All reports to AIV during September, October, and November 2021 were examined. There were 7 689 cases overall. 10 841 edits were made while reports were waiting, of which 8 020 were both reverted and made by a user who got blocked.

Note that this next chart's horizontal axis is the minimum number of edits; that is, the leftmost bar means that there are 1271 cases where 2 or more edits were made, and there were 157 cases where 10 or more edits were made (an average of 14 and 1.7 cases per day, respectively).

Worst cases (first link label is number of edits while report active, second link label is response time; all of these ended in a block):
Reverted edits: 85 (2h), 79 (31m), 71 (1h), 69 (3h), 63 (2h), 55 (51m), 51 (2h), 45 (1h), 43 (14m), 43 (5h), 43 (1h), 38 (3h), 37 (2h), 37 (14m), 35 (26m), 34 (2h), 33 (16m), 33 (1h), 33 (5h), 33 (1h), 32 (30m), 32 (51m), 32 (5h), 30 (38m), 30 (2h), 29 (19m), 27 (27m), 27 (31m), 26 (39m), 26 (31m),
All edits: 89 (2h), 86 (5h), 84 (1h), 83 (37m), 79 (31m), 70 (3h), 63 (2h), 58 (51m), 51 (2h), 45 (1h), 43 (1h), 43 (14m), 38 (3h), 37 (38m), 37 (14m), 37 (2h), 36 (26m), 34 (2h), 33 (1h), 33 (5h), 33 (5h), 33 (16m), 33 (1h), 32 (30m), 32 (51m), 30 (2h), 30 (39m), 30 (19m), 28 (2h), 27 (31m), 27 (37m), 27 (27m), 26 (5m), 26 (31m), 25 (5h)

Methods

I read the page history of AIV; for every edit removing text, I parsed reports from the text that was removed. For each report, I fetched the contributions of the reported user starting when they were reported and ending when the report was resolved. Reports could either be resolved by blocking the reported user, or by adding a comment with {{AIV}}. If both occurred (a block and a comment), the block took precedence; if multiple comments occurred, the final one took precedence.

The source code used for this project is at https://git.sr.ht/~enterprisey/aiv-analysis-jan-2021/tree. Both the program to collect data (written in Rust) and the scripts to analyze the data (written in Python and Rust - see analyze/ and python_analyze/) are there.

Things to do in the future?

scatter, per hour of day, # reverted edits against median response time, 24 data points
total pageviews during "revert wars"?
do any "important" articles get hit? vital articles? rank N or top X% by pageviews?
what % had revdel invoked
redo all this for RFPP
Edit rate histogram (just worst cases, maybe?)