User:Tzusheng/sandbox/Validation Study
The validation study of Wikibench.
Background
[edit]This validation study is part of the broader research that aims to support Wikipdians to evaluate AI systems, such as ORES and LiftWing, by curating an evaluation dataset that Wikipdians can use to compare against AI's predictions.
To support the curation of the dataset in the context of edit quality, we developed Wikibench, a wiki plug-in that enables Wikipedians to label whether an edit is damaging or saved with good or bad faith while patrolling recent changes. Wikibench also has several features that facilitate discussion when Wikipedians disagree on the label. During our pilot testing, Wikibench received overwhelmingly positive feedback from Wikipedians.
The purpose of this validation study
[edit]The last mile of this pilot testing is to validate whether the data and its label curated using Wikibench indeed reflect community consensus. Therefore, as shown in the table below, we sample a set of edits from Wikibench's dataset and invite experienced Wikipedians to collectively discuss and decide what the "correct" label of these edits should be. We'll compare the labels Wikipedians enter in this table to Wikibench's labels to understand how many of them are identical or different and to know the effectiveness of Wikibench in terms of "correctness".
How to participate
[edit]For invited Wikipedians, please enter your label in the column of your username. For edit damage, the label is either damaging or not damaging. For user intent, the label is either good faith or bad faith.
After labeling all the edits, please be bold to enter or update the consensus based on your and others' labels in the consensus column. Whenever there are disagreements, please discuss them on the talk page. Note that you don't necessarily have to wait for others' labels to enter the consensus.
The study completes when the consensus columns are filled with labels that are reflective of the consensus of the five participating experienced Wikipedians. Note that consensus is not necessarily the majority label.
Study duration
[edit]This validation study lasts about a week, from July 14 to 22, 2023. It might be extended for another week if Wikipedians find the extension helpful for building consensus.
Compensation
[edit]We'd provide $90 USD to compensate for your time, experience, and contribution. As discussed in the village pump and concluded here, this research is not the paid editing that Wikipedians and WMF are concerned with. However, you may still consider adding the following userbox for disclosure if you decide to accept our compensation.
{{User:Tzusheng/Userboxes/WikibenchDisclose}}
 | Per Terms of Use, this user discloses being paid by CMU as part of a study for contributing to Wikibench. |
Selected edits for validation
[edit]| Diffs | Edit damage | User intent | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Title | Diff IDs | Zippybonzo | Skarmory | Bluerasberry | Matthewrb | Schminnte | Consensus | Zippybonzo | Skarmory | Bluerasberry | Matthewrb | Schminnte | Consensus |
| Adjacent | 645892572/677829222 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | good faith |
| Elephant goad | 667271580/677974402 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | good faith |
| Indian name | 680080662/680081690 | not damaging | not damaging | ๐ damaging | Not damaging | not damaging | Not damaging | good faith | good faith | ๐ good faith | Good faith | good faith | good faith |
| Empire (2015 TV series) | 681052757/681086400 | damaging | not damaging | ๐ damaging | Not damaging | not damaging | not damaging | good faith | good faith | ๐ good faith | Good faith | good faith | good faith |
| List of awards and nominations received by Carly Rae Jepsen | 682416765/682418557 | damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| T. A. Shahid | 664715336/683209659 | damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Ollie Hayes | 680497781/683511584 | damaging | not damaging | ๐ damaging | Not damaging | not damaging | not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Panathinaikos B.C. | 684796093/684882507 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Mr Green (company) | 677934765/685548844 | Not damaging | damaging | ๐ not damaging | Damaging | damaging | not damaging | Good faith | good faith | ๐ bad faith | Good faith | good faith | Good faith |
| Montreal Canadiens | 691789091/691911798 | damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Tales of Mystery and Imagination (Nocturnal Rites album) | 618199239/692384164 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Bad faith | good faith | ๐ bad faith | Good faith | bad faith | bad faith |
| Ilocano language | 694270988/694271323 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Bad faith | bad faith | ๐ bad faith | Bad faith | bad faith | Bad faith |
| Masters Champions League | 695181325/696058558 | Damaging | not damaging | ๐ damaging | Damaging | damaging | damaging | Bad faith | good faith | ๐ bad faith | Bad faith | bad faith | bad faith |
| List of Telugu films of 2015 | 698373777/698392149 | Damaging | not damaging | ๐ damaging | Damaging | damaging | damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | good faith |
| Betty Holberton | 691777400/699667798 | Not damaging | not damaging | ๐ damaging | Not damaging | not damaging | not damaging | Good faith | good faith | ๐ bad faith | Good Faith | good faith | good faith |
| User talk:Bishnu Rasaily | prev/703976291 | Not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Soldier Fuel | 704809356/705203406 | Not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Ousmane Dembรฉlรฉ | 705814739/705815021 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | bad faith | ๐ good faith | Good faith | bad faith | bad faith |
| Heather O'Rourke | 705812016/706027789 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | good faith |
| List of presidents of the United States by age | 708655010/708695027 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | bad faith | ๐ good faith | Bad faith | bad faith | bad faith |
| Charlotte Douglas International Airport | 713303042/713770261 | Not damaging | not damaging | ๐ damaging | Not damaging | not damaging | Not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Melt | 710265936/714932920 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Sam Kim | 715352163/715353867 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| 2016 Fort McMurray wildfire | 719225065/719236236 | Damaging | not damaging | ๐ damaging | Damaging | damaging | damaging | Good faith | good faith | ๐ good faith | Bad faith | good faith | Good faith |
| St. Mark's School of Texas | 719347634/719359416 | Not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Joi Lansing | 715632333/719925637 | Damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Robert Wadlow | 720771351/720797296 | Not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| N.O.R.E. | 722177677/722563742 | Not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | not damaging | Good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Nelson Story | 719484034/723407464 | not damaging | damaging | ๐ not damaging | Damaging | damaging | damaging | good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| Talk:Bollywood films and plagiarism | 670938590/726418154 | not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| 2016 Novak Djokovic tennis season | 724757924/726787424 | not damaging | not damaging | ๐ damaging | Not Damaging | not damaging | Not damaging | good faith | good faith | ๐ bad faith | Good faith | good faith | good faith |
| Witold Sobociลski | 726553825/730265683 | not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |
| The Adventures of Super Mario Bros. 3 | 723218057/731155214 | damaging | damaging | ๐ damaging | Damaging | damaging | Damaging | bad faith | bad faith | ๐ bad faith | Bad faith | bad faith | bad faith |
| State Road 10 (Serbia) | 735558460/735558653 | not damaging | not damaging | ๐ not damaging | Not damaging | not damaging | Not damaging | good faith | good faith | ๐ good faith | Good faith | good faith | Good faith |