Wikipedia:Research help/Pilot report

Summary
Many readers are unfamiliar with how Wikipedia works and how to use it responsibly. To improve reader understanding of research using Wikipedia, we added a link to a Research help page in the reference section of 10,000 articles under WikiProjects Medicine and Military history.

• During the trial period there were 24,862 pageviews (228/day) to the Research help page, of which 12,838 (133/day) came from the targeted Wikiprojects.
• At that rate, expansion to all English Wikipedia articles would approach 70 million annual pageviews, declining somewhat over time due to exposure and familiarity.
• There were six on-wiki discussions about the page and the link to it, attended almost entirely by highly experienced editors, demonstrating a generally negative response.
• We separately collected 43 survey responses from people who visited the page.
• 75% of the survey respondents thought that a small link on Wikipedia articles to the Research help page would be helpful to Wikipedia readers.
• Closer analysis of survey responses shows far stronger support from those with under 100 edits (96% support), and far more opposition from those with 100+ edits (53% oppose).

It is clear from our analysis that newer and non-editors liked this intervention more by far, while most of the opposition to the page and the link came from experienced editors.

This suggests a need for further discussion with the community to build consensus for future experimentation and also about which audience's opinion to prioritize in interface changes that are reader and new-editor focused. We suggest a stage 2 pilot that reasonably broadens the number of pages viewed in both number and diversity in order to collect better data, while also mitigating frustration from experienced editors by having the link only appear to new and non-editors. Community consensus will be vital to continuing experimentation, and we view it as well worth future engagement.

Motivation

The Wikipedia Library provides support for improving research access, literacy, and effectiveness. While we are best known for providing free accounts for experienced editors to expensive paywalled databases and publishers, an increasing interest has been reader engagement with the research materials cited by our community.

As research demonstrates, many readers are entirely unfamiliar with even the basics of how Wikipedia works and how to use it intelligently. In one demonstrative example, a Rutgers study of undergraduate students revealed that the majority of them believed that reference footnotes were presented in order of importance not order of appearance. There is a vast divide between practice and pedagogy--many teachers instruct students never to use Wikipedia, but surveys show that nearly all students do.

“College students use Wikipedia frequently, despite educators’ highly divided opinions about it... This study found that first-year students are uncertain about the variety of ways to use information sources like Wikipedia.”

— Getting a ‘quick fix’: first-year college students’ use of Wikipedia, ‘’First Monday’’

“Wikipedia is increasingly becoming the go-to reference resource for the newest generation of students. However, many students do not know about the problems (e.g., vandalism) associated with this tool other than ambiguous warnings from librarians and faculty who say that it should not be used for research.”

— Using Wikipedia to Teach Information Literacy, ‘’College & Undergraduate Libraries’’

This raises fundamental issues of critical literacy in a ubiquitously used resource and the conflicting attitudes towards whether readers should trust it. We believe there may be a role for the Wikimedia community to play in addressing these concerns by explaining how Wikipedia works (and doesn't). To this end, we designed a pilot project with the goal providing readers with a basic understanding of how to use Wikipedia effectively, responsibly, and critically. The project would distribute basic digital and research literacy information about Wikipedia through a link in the References sections of ten thousand articles.

Plan

The Wikipedia Library Team wrote up our plan in a proposal, and then crafted the Research help page to address a broad public audience. The focus audience comprised students, instructors, and scholars who had misinformation or doubts about the reliability of Wikipedia. Our project operated on the assumption that if we could inform and/or allay these concerns, then we would address the majority of common unknown or misunderstood aspects of using Wikipedia for research. These misunderstandings sustain the resistance, confusion, and conflicting attitudes in schools and in the broader world towards Wikipedia as a general reference or starting point for information.

The Research Help page answers key questions, offers tips, and directs specialists to targeted portals about Wikipedia.

FAQ

• When is Wikipedia useful for research?
• Should I cite Wikipedia?
• How reliable or accurate is Wikipedia?
• How are Wikipedia articles structured?
• How can I know a Wikipedia article's quality?
• What if I can't access sources cited in a Wikipedia article?
• What if Wikipedia doesn't have the information I need?

Tips

• Read tips about navigating and finding content on Wikipedia
• See tips for finding the sources cited in Wikipedia
• Go to our reference desk where Wikipedia editors can help you find information
• Find a librarian near you whom you can ask for help

Specialist portals

• Are you an educator or librarian?
• Are you a scholar or expert?
• Do you work in cultural heritage?
• Are you a Wikipedia editor?

Evaluation

We collected data to answer 4 key questions about the reader experience:

• How often was the Research help link clicked on?
• Did readers of the Research help page find its content useful?
• Did they think that a link on articles to the page would be helpful to Wikipedia readers?
• Where did readers of the page think such a link should be located?

Extensive discussions occurred on-wiki during the course of the pilot. This qualitative data responded to similar key points from the editor point of view:

• Did Wikipedia editors support the idea of a Research help page?
• Did editors support the page’s current content and design?
• Did editors support the inclusion of a link to this page?
• Where did editors believe such a link should be placed?
• By what process did editors believe the page's content should be disseminated?

Method

To test the effectiveness of these pages with readers, we solicited consensus from two Wikiprojects, Wikiproject Medicine and WikiProject Military History to place the template on the articles within their topic area. We did so under the condition that this was a temporary and limited experiment and that any broader test would require broader community consensus. We chose to focus on two different WikiProjects, in part because we wanted to understand if readers of humanities and science articles on Wikipedia would respond to the links in the reference sections differently.

• 7 editors !voted on WikiProject Medicine: 6 Support, with no Oppose, and 1 Neutral
  • Supporters thought it was a "great idea", "lovely idea", or "seems like a good idea" with its "very unobtrusive link".
  • They also noted the importance of a gradual rollout to articles alongside "measureable outcomes"
• 14 editors !voted on WikiProject Military History: All Support
  • Supporters thought it was a "good idea" and "a positive link to learning"

A bot was approved to put small links that said "Using Wikipedia for Research" under the ==References== section header on approximately 5000 of each WikiProject's pages (based on alphabetical order starting with the beginning of each project's category).

To identify from where users were arriving to the Research help page, we created a variable within Template:Research help which split traffic through four redirects:

• WP:Research help/Med, for Medicine pilot articles
• WP:Research help/Mil, for Military History pilot articles
• WP:Research help/Gen, versions of the template added by volunteers outside of our pilot
• WP:Research help/WP, which allows tracking of links within the Wikipedia namespace

On March 24, 2016, User:Rich Farmbrough made a minor change to the link text from “Using Wikipedia for Research” to just “Wikipedia and Research” with this revision. At the time, we had collected over two weeks of pageview data on the impact of the original language - and with community conversations about the future of the template, we did not challenge the revision, so further data was collected with that text in place.

During the trial, the link template was nominated for deletion. The result was no consensus. Later, there was a discussion at Village Pump concerning the template and the result was to no-include it, effectively ending the pilot early.

The link was present on WikiProject pages from December 17, 2015 through April 3rd, 2016: the bot ran its trial of 100 articles in December and was fully approved in February to add the full 10,000 pages. We then analyzed pageviews, survey data, and on-wiki discussions. We further broke down the analysis by whether feedback was on a) the page, b) the link to the page, or c) the process of the pilot itself. Finally, we broke down the analysis by cohort of new or non-editors (under 100 edits) vs. experienced editors (100+ edits).

Results

Pageviews

From March 7th to April 3rd, links were in place on ~5,000 pages each in the topics of Medicine and Military History. There were 12,838 pageviews (459/day) [Query]:

• Wikipedia:Research help/Med: 8,109 (290/day)
• Wikipedia:Research help/Mil: 1407 (50/day)
• Wikipedia:Research help/WP: 1709 (61/day)
• Wikipedia:Research help/Gen: 145 (5/day)
• Wikipedia:Research help: 1468 (52/day)

Far more views came from the medical articles than from military history (though there wasn't a significant difference in their opinions on the page).

For the broader period from the initial December 17 bot approval to the end of the pilot, there were 24,862 pageviews (228/day) Query:

• Wikipedia:Research help/Med: 12,258 (112/day)
• Wikipedia:Research help/Mil: 2,245 (21/day)
• Wikipedia:Research help/WP: 6,185 (57/day)
• Wikipedia:Research help/Gen: 1,209 (11/day)
• Wikipedia:Research help: 2,965 (27/day)

Survey

The page included a link to a reader survey. The survey received 43 responses between November 1, 2015, and March 30, 2016 (although not every respondent answered every question). We made no substantial changes of content to WP:Research help during that window of time. Over half of the respondents self-identified as students, teachers, researchers, or librarians.

On-wiki discussions

There were six main venues for on-wiki discussion: WikiProject Medicine, WikiProject Military History, the Bot Approvals Group, Templates for Discussion, Village Pump (Technical), and the Research Help talk page. These discussions saw 88 unique participants across all venues, with some taking part in multiple discussions. Participants tended to be very experienced, with an average edit-count of 65,302 on English Wikipedia. Only one unregistered user took part in the discussions and only two users with fewer than 500 edits participated.

The page

Quantitative

Of the 43 survey respondents:

• 50% (20) found the page to be particularly useful while 35% (14) did not.
• All of the 14 respondents with high-school or a not-yet-complete undergraduate degree thought the page would be helpful to readers.
• Of the 43 respondents there were 3 Librarians, 14 students, 7 educators, and 5 researchers (so 29 of 43 were in our target group)
• 72% (28) would recommend the page to a student.
• 59% (23) would recommend the page to a teacher.
• Most respondents previously knew some or most of the information.

Qualitative

Survey

Survey results show that established editors (those with 100+ edits) and new or non-editors (under 100 edits) both responded positively to the page content and the information it provided about using Wikipedia. Out of 15 responses, 5 were positive and none were negative about the page content.

Both groups offered suggested improvements to the page design, but in opposite directions. One young non-editor requested “some more information that isn’t links to other pages”, whereas an established editor suggested that “we could simplify the content even more” and another thought “it seemed too long, too wordy”.

Several survey responses affirmed one of the page’s primary messages, that: "Wikipedia is only as reliable as the sources” and “Wikipedia is a good place to start research”.

On-wiki discussion

Many agreed that the portal was and should be primarily targeted to readers and new editors (“It is not directed to editors, but to readers of Wikipedia who are not accustomed to analyzing sources in the same fashion”).

Commenters were more positive than negative about the objective of the page (“A pilot to see if we can help readers find their way into deeper knowledge is not a bad thing to me, and is aligned with our mission”). Some editors suggested the page should be more tailored to the content of the articles in which it was linked; others commented that it should be limited to specific article types where readers “are more likely to benefit” (e.g. medical articles).

The on-wiki discussion of page content was mixed but generally negative (one user described it as a "propaganda" page that “might mislead the readers in believing that Wikipedia is more reliable than it really is”).

Of those commenters that did discuss the page, none expressed positive feelings about its design while several were concerned about depth or length (“condense the page down a bit as well as it needs to be as short as possible to get the message across”).

The link

Survey

75% of all survey respondents thought a small link on Wikipedia articles would help readers.

When asked where they preferred the link to be placed:

• 33% (13) thought the link would be most helpful to readers if placed in the left-hand menu
• 28% (11) preferred the references section
• 28% (11) said the top of articles
• 5% (2) suggested reference tooltips
• 8% (3) made other suggestions (such as the talk page or nowhere).

Some expressed positively that the link exposed them to The Wikipedia Library and its projects, of which they had previously been unaware. Others expressed concerns that the link did not belong in article space and shouldn't exist at all.

When asked where the link would be too obtrusive:

• 73% (22) said the top of articles
• 57% (17) said in reference tooltips
• 40% (12) said in the references section
• 17% (5) said in the left-hand menu
• 10% (3) gave other responses.

Respondents were allowed to make more than one selection for this question.

On-wiki discussions

On-wiki discussions showed negative feedback about the link (over 10 negative comments for every positive).

Pointer placement was particularly contentious, with 92 negative and 13 positive comments; most of the negative comments arose from the Templates for Deletion discussion, with editors noting that the page is “meta” and contributes to “well meaning clutter”.

The link design was of less interest but was also disliked, with 18 negative and 3 positive comments: (“On longer articles, it's likely to be lost in the general goop of navboxes, Commons links, portals and external links at the bottom of the page if not made even more obtrusive than it already is”).

Some editors expressed a desire to be able to dismiss the template once they had seen it, or to not see it at all when signed in:

“Templates such as {{Citation needed}} and {{Refimprove}} are perpetually useful for readers (to indicate that the verifiability of a statement or article is poor), but most readers will not use the page more than a few times”)

Various alternative placements for the link were suggested, including the left sidebar and the article talk page (“[I] don't believe a link to a project essay in the content of a main space article is a good idea ever”). Technical solutions were also proposed, such as adding the pointer into {{reflist}}.

Problems with alternative link placements were also noted (“Adding it to the talk-page entirely defeats the purpose and makes it irrelevant” [to the readers and new editors who were the primary targets]; “There's no good evidence that regular readers look at the sidebar, and secondly, the sidebar doesn't even exist on the mobile site, which is where more than 30% of pageviews happen”).

The process

Bot-placement of the link received support from the WikiProjects directly affected by the pilot (WP:WikiProject Military History and WP:WikiProject Medicine):

“The trial was discussed centrally and approved - I certainly also saw the discussion at WT:MED and think that a four-week trial is perfectly reasonable. It's not as though the plans were only on display at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying beware of the leopard.”

In broader on-wiki discussions, many established editors provided negative feedback on the process by which the link was added to articles, expressing concern that inadequate consensus-building opportunities had been offered prior to the larger bot run of 10k articles. Noting WP:LOCALCON:

“None of this suggests that the community as a whole had a thorough discussion of this unusual template prior to rollout.”

Some viewed this as part of a perceived broader trend of WMF failing to engage with community concerns:

“I was actually surprised to find out that this was yet again a WMF-backed project, after Flow, Gather, SuperProtect, etc. they still fail to understand that communication is key”.

Analysis: non/new editors vs. experienced/prolific editors

Closer analysis shows even stronger support from those with 0-99 edits (n=25), and more opposition from those with 100+ edits (n=16).

75% of survey respondents overall thought that a small link on Wikipedia articles to the research help page would help Wikipedia readers (30 said yes, 10 said no). All 13 respondents with 1-99 edits and 9 of 11 with no edits said it would be helpful to readers. 9 of 11 who thought it would not be helpful had 100+ edits.

Page Usefulness

Under 100

23 respondents with under 100 edits who rated the page's usefulness between 1 of 7 (with 7 being most useful). Only 1 respondent with under 100 edits thought it was not particularly useful while 17 thought it was particularly useful.

100+

13 respondents with 100+ edits rated the page's usefulness. 10 respondents with 100+ edits thought it was not particularly useful while only 3 thought it was particularly useful.

Would recommend

Respondents were asked if they would recommend the page...

To a student

Of 22 respondents with under 100 edits, only 1 was not at all likely to recommend it to a student while 13 were very likely (on a scale of 1-3, with 3 being most likely):

Of 14 respondents with 100+ edits, 8 were not at all likely and only 1 was very likely to recommend it to a student (on the same 1-3 scale):

To a teacher

For recommending to a teacher (on the same 1-3 scale) those with under 100 edits were equally split here with 8 not at all likely to recommend to a teacher and an equal 8 very likely to:

For recommending to a teacher by those with 100+ edits, 9 were not at all likely to recommend it and only 1 was very likely to:

Quotes

Under 100

There were many more positive quotes from respondents with under 100 edits:

“ It took like a decade for me to see that such an important page exists. ”

“ It explicitly lays down some of the critical integral rules of where Wikipedia means to stand as a free public purveyor of general knowledge. ”

“ [It was] clear and short, probably a good intro for high school kids and students. ”

“ [It] answered all questions I would pretty much have in a short and to-the-point manner. ”

“ [It] had links to other pages for further information. ”

“ [It was] well written and concise....especially...how it is stressed to do primary research. ”

There were only a few negative quotes from respondents with under 100 edits:

“ I'm not sure if it would answer all questions someone might have. ”

“ There wasn't a lot of information on that page, it was mostly just links to other pages with answers to various questions. ”

100+

There were many more negative quotes from respondents with 100+ edits:

“ Layout sucks. Articles being spammed. ”

“ It's just spam and low-value info. ”

“ [It's most surprising] that it, a pointless feature, started showing up in articles. Although, that is the classic Wikimedia Foundation way of doing things, so I really shouldn't have been surprised. ”

There were only a few positive quotes from respondents with 100+ edits:

“ I like it because it is a disclaimer clarifying what the content can be used for. I am sick to death hearing how horrible Wikipedia is for research. I get criticized all the time for being an editor." ”

“ It's potentially useful for new editors to shortcut the learning curve. For me, nothing new or useful - sorry!" ”

Link placement

Under 100

Respondents with under 100 edits most preferred the top of the page or the reference section for the link location.

100+

Respondents with 100+ edits most preferred the left-hand sidebar for link placement, and also found it the least obtrusive option. These established and highly active editors also reacted negatively to the placement of the link in article space at all. They were more likely than non-editors to suggest removing the pointer entirely or moving it to the talk page.

Conclusion

It is clear from our survey analysis that most respondents found the page and link useful--but new and non-editors liked it more by far, while most of the opposition to the page and link came from experienced editors.

The disparity in support between those at whom the pilot was targeted--readers and new editors--versus the reception it received from experienced editors calls for further discussion and iteration of the experiment.

Stage 2 pilot proposal

Reasonable readers and editors will react to this report in different ways. Some will say that our conclusion is definitive: after all, 96% of the target audience supported it! On the other hand, detractors will note that this group was only around 20 people out of the 130 who discussed it in total.

Selection bias may be seen in the survey if editors who liked the page were more apt to tell us so in the survey (whereas the less impressed just ignored the page). On the other hand, editors who are vehemently opposed to the link/page may have filled out the survey all the same just to share their displeasure. This potential bias cuts both ways.

The sample size of survey respondents may be deemed too small to draw strong conclusions, and we actually support that view. We need more data. The only way to get that data from a broad audience, however, is to put the link back up.

Our strategic approach moving forward is to design and seek community consensus for a stage 2 pilot which:

• Places the link on a percentage of all English articles (1% to be precise), using a randomized approach
• Shows the link only to non-logged-in users (or those with under 100 edits if technically possible), to minimize disruption to experienced editors

Doing so will give us a diversity of feedback outside of just the two studied Wikiprojects. It will increase the pool of readers 5x from 10,000 articles to 50,000 articles. It will allow us to collect sufficient data to draw stronger conclusions. We will also use a redesigned and even more rigorous survey to tease out finer and clearer distinctions in the data such that we could strongly support clear decisions after the pilot. Having had the benefit of this "local" trial, we will be in a position to involve the editing community broadly in a discussion about the pilot design, based off of concrete feedback rather than opinion or supposition. We will be able to mitigate the most off-putting aspects of the pilot concerning experienced editors. Finally, we will generate further interest in the goal of increasing reader research literacy of Wikipedia.

Next steps

We'll continue to study, iterate, and experiment, though not necessarily in the exact order of the following components.

We need to present the above pilot results and seek feedback on the report. We need to explore tweaking approaches for the link, such as using different designs or placements. In particular we should look into the ability to dismiss the link for logged in users, or create a link that only appears for non-logged-in users.

We should further workshop the page content and design to address concerns over time, continuing to iterate on this throughout the experiment. We are soliciting research on the portal and other literacy tools in general: two academics have committed to doing targeted focus-group research in the summer and fall of 2016 to improve the page itself using existing scholarship. We should also test small-scale walkthroughs to identify pain points in the page design.

We must refine the process for suggesting expansion of the number of articles affected by the pilot. We may need to hold a larger RfC, or at least a more broadly advertised discussion, in order to gain consensus for a second rollout applied to significantly more English pages.

Lastly, we want to disseminate this experience to other language communities who may be interested in similar interventions towards building reader and new editor research literacy.

Please leave comments on the report and next steps on the talk page.