Wikipedia:Help Project/Usability Testing 1

A series of usability tests were carried out by the wub in the Summer of 2012, as part of the help pages community fellowship. These are the results and observations from those tests.

Participants

Name	Age	Gender	Wikipedia reading	Wikipedia editing	Occupation	Other notes
J	25	M	Every day	Couple of typo fixes	Student
E	26	F	Every day	None	Student	Also uses Hungarian Wikipedia occasionally. Thought about editing in the past, put off by code.
H	23	F	Few times a week	None	Teaching assistant
A	52	M	Few times a week	None	Engineer
M	48	F	Few times a week	None	Teacher

Introductory script

(It may be a little strange since I know you, but I'm going to be reading from a script. This is just to make sure I don't forget anything, and make sure everyone participating gets exactly the same instructions)
Thank you for agreeing to take part in this.
First of all:
During the tests, I'll be using a piece of software to record what's happening on the computer screen, and also to record your voice through the microphone. This will be for me to refer to later when I write up my results. The recordings won't be shared with anyone else. I just want to check you're okay with this.
If they say yes
Ok thank you. So the purpose of these tests is to collect feedback on Wikipedia. Basically I'll ask you some questions and then watch you use the website.
This will take about 30-45 minutes.
Your job is really easy, you just have to be yourself and act as you naturally would. As you interact with the site, please be honest with your positive and negative thoughts. I didn't design any of these pages, so don't worry if you have negative feedback, it's not going to hurt my feelings!
It's important to remember that you are not being tested, the Wikipedia pages are being tested.

Background questions

Before we start the tests proper, I just want to find out a bit more about your past experience with Wikipedia.

How often do you visit Wikipedia?
Do you ever use Wikipedia in any languages other than English?
You said you come to Wikipedia XXX often. What are you usually looking up/doing? How has your experience been in the past?
Have you ever edited Wikipedia at all?
(Non-editors) Are there any particular reasons that you haven’t edited in the past? Can you tell me what you know now about the process of editing?
(Editors) You mentioned that you have contributed to Wikipedia before. Please tell me about those experiences.

Think aloud and starting

Next I’d like to have you attempt a few tasks on Wikipedia.
There’s one thing that I would like you to do differently. Please think aloud as you carry out these tasks.
For example, if you are reading something read it out loud and feel free to say anything that comes to your mind as you read - for example "that’s interesting" or "what does that mean?" Also, talk out loud while you’re doing a task, so I can understand what you are thinking and doing. For example: "Now I’m going to try to use the search engine")
If you get to a point where you would naturally stop working, let me know. I’ll let you decide a task is complete, whether you find what you are looking for or not. Just tell me when you’re finished.
We’re going to be focusing on navigating through the website and editing entries. For the tasks that involve editing, we will be previewing your changes but we won’t actually be permanently saving any changes to Wikipedia. I'll walk you through that when we come to it.
Do you have any questions before we continue? Okay, let’s go to the Wikipedia website.
Take them to English Wikipedia main page
Log in to test account

Task 1: Finding help

My work is attempting to improve the Help pages on Wikipedia. So imagine you want to get involved in editing Wikipedia. Where would you look for help starting out?
The main purpose is to see where they end up. Expected routes: Help in left sidebar, typing "help" or similar in search and clicking hatnote.
If they go to the help desk, let them try to write out a question and preview, but stop short of posting it.

Observations

J

"Why is Help under 'Interaction'? I think it should be somewhere more obvious." Clicks on Help in the sidebar. -> Help:Contents
"Getting started sounds right" -> Help:Contents/Getting started
"Wow, this is a lot of links" Reads for a while. "I'm not sure which of these is better, Introduction or Tutorial? I'll go for Introduction." -> Help:Getting started
"Oh, now this page is called Getting started as well? I guess I should click "Introduction — What is Wikipedia?", but I thought that's where I was." -> Help:Introduction
Reads page. "Hmmm, the 'Sandbox' sounds interesting. Maybe I should try that." -> editing Wikipedia:Sandbox
Types "hello world", and after some searching finds the button to preview the page.
I suggest we move on if he's happy to.
"Well I feel like I should read more, but it just seems to be going round in circles. And editing in the Sandbox was not too hard. So yes, I'm okay to move on."

E

Clicks on "Help" in the sidebar. -> Help:Contents
"This (Getting started) looks like what I want." -> Help:Contents/Getting started
After reading links for a few seconds. "This isn't what I expected. I'm going to try going back to the other page." Hits back button -> Help:Contents
"Ok, this time I'll try 'Editing Wikipedia'" -> Help:Contents/Editing Wikipedia
Reads. "I don't think I need the 'Introduction to the project' because I know quite a bit about Wikipedia, I just want to learn about editing. 'Contributing to Wikipedia' sounds more useful." -> Wikipedia:Contributing to Wikipedia
"Oh, these are videos. Cool." Watches File:Edit Button.ogv and File:Great Feeling.ogv. "Well, they were nice, but they didn't really tell me much about how to edit. Just said 'click edit and save'. Maybe I should just try that, rather than all this reading."
At this point I asked if she wanted to move onwards to tasks that involve editing, and she agreed.

A

Typed "editing" into the search box and pressed Enter, he was taken to Editing. "I guess this is about editing generally, not Wikipedia."
Eventually noticed the hatnote "Wikipedia:How to edit a page", which actually redirects to -> Help:Editing
Read the whole page (but did not watch the video).

M

Clicks on "Help" in the sidebar. -> Help:Contents
Clicks almost immediately on 'Getting started' -> Help:Contents/Getting started
"I'm not sure which of these to click. I don't think it's this one ('Who writes Wikipedia'), that says it's for readers and I'm supposed to be learning to edit. I'll try 'Introduction to Wikipedia'." -> Help:Getting started
"Oh, now it says Getting Started again. And there's another Introduction link. Well I'll try this one." -> Wikipedia:Introduction
Read the first page and made a successful Sandbox edit (in preview). -> Wikipedia:Introduction 2
"There's quite a lot on this page. 'Learn more about editing pages' sounds like it will be helpful." -> Help:Editing.
"This page is long. Is that a video though? Maybe that will be easier." -> watches File:Wikipedia video tutorial-1-Editing-en.ogv
"Well it was helpful, but her Wikipedia looked different from mine." (it was monobook) "Also it was too blurry to read the text properly and the audio was very out of sync in places."

H

Clicks on "Help" in the sidebar. -> Help:Contents
"Lots of links here, I'll try this one." -> Help:Contents/Getting started
"This has even more links! *looking around* What are these ones over here? *reading tutorials box* Some of these are very strange. 'Grutness's guide to stubbing' - what the heck does that mean? *reads for a bit longer* I think I'll go with "Introduction to Wikipedia, that sounds simple." -> Help:Getting started.
Reads. "If you would like to edit an article, the Basic tutorial will show you how. That sounds like what I want then." -> Wikipedia:Tutorial
"This looks a bit ugly compared to that other page."
Progresses through Tutorial up to Citing sources tab. "This is longer than I thought. And every page seems to get longer and more complicated. I kind of understood the other pages," (and she made successful edits in the Sandboxes) "but this is too confusing. It's a lot to learn all at once."
It takes her some time to open the "Visual inline citation guide", the show link is not very visible on a blue background. And she finds the text in there confusing.

Notes

We were already aware that Help:Contents is tricky, especially Help:Contents/Getting started. This has a prominent placement and appears the most obvious link to follow (all 4 subjects who reached Help:Contents clicked on this almost immediately). However they were then puzzled by the large number of links. 3 of the 4 eventually went to Help:Getting started (with two of them commenting on the confusingly similar names, made worse by the fact it is linked as another "Introduction"!). These observations support the plan to significantly improve Help:Getting started, and link it directly from the main help page, rather than having to go through an extra step.

H's experience suggests the Tutorial could use some work. As she pointed out, some sections do get rather long and complicated. Maybe we could somehow make it more obvous that you don't need to complete it all in one sitting. It would also make sense to improve and standardise the hatnotes from article space, as it appears they may be a significant route into the help system.

File:Wikipedia video tutorial-1-Editing-en.ogv was new to me. It's nicely put together with good explanations, but suffers from the poor video and audio quality. In addition the interface, particularly the editing toolbar, has changed a bit since the video was made. This is an inherent problem with video tutorials, they are hard to update.

Task 2: Help on references

Let's look for help on something a bit more specific now. You know those footnotes Wikipedia articles have with references? (show them if they don't understand) Try and find out how you would add one to an article.
Watch where they go, this should be interesting :-) Want to see how they search for specific info. Stop them after a while if they aren't getting anywhere (or they attempt to read the whole of Wikipedia:Citing sources) and move on.

Observations

J

Returned to Help:Contents
"Ah, I thought so, there's a page here about 'links and references'." -> Help:Contents/Links
Reads page. "How, why, what. I quite like that. Seems I want 'how'. " -> Wikipedia:Citing sources
"Ok. There's a link to something about footnotes. I thought it would be on this page. Maybe it is, but this page is kind of long. I'll see if the other one is any shorter." -> Help:Footnotes
"Doesn't look much shorter." Scrolls down fairly fast. "But at least this seems to have all the code. Yeah, I think this is the page I want." Reads the first four sections.
"Hmmm... I think I understand. Would like to try it out though."

E

"It was hard to find things from the Help menu before. Maybe I should just try searching for it."
Types 'footnotes' into search box and presses Enter -> Note (typography)
"Aha, 'For the usage of footnoting on Wikipedia, see Wikipedia:Cite sources and Wikipedia:Footnotes.'" The second one must be what I want. -> Help:Footnotes
Reads a bit. "I'm not sure I get it. Do I have to type these 'tags' in when I edit? Is that HTML? It looks a bit like it. It doesn't seem very clear."

A

"I think there was a section on that page (Help:Editing) about adding references." Goes back to -> Help:Editing
Reads. "Yep." Clicks 'Adding references' in ToC. Then clicks -> Wikipedia:Referencing for beginners.
Reads. "Oh, there's a video. Maybe I should watch that instead." -> watches File:RefTools.ogv
"Okay, that actually looked much simpler than I was expecting."

M

Returned to Help:Contents
"I don't know, footnotes, I don't see anything about those here. Huh, what's this 'Search Help' button... oh, I type in there I suppose (points at box above).
Types "footnotes" into box, and clicks 'Search Help'.
First result is Help:Footnotes. "Good, that looks as if it's what I want." Clicks. -> Help:Footnotes
Reads, clicks on picture to expand. "This is kind of confusing. I'm not sure what all the stuff about tags means. And the page is so long."

H

H had already found and read the tutorial page on Referencing, so we skipped this task.

Notes

Again, evidence of how crucial well-placed hatnotes in the article namespace are, for people who use the default search engine. A shame there is also tension to avoid self-references such as this.

Only one subject reached Wikipedia:Referencing for beginners, and unfortunately that is the only place that RefTools (by far the easiest way of doing citations for beginners) is explained. Help:Footnotes doesn't even mention the Reference button in the toolbar either, assuming people will type out the code by hand (or more likely copy and paste from elsewhere.)

Task 3: Editing wikitext

Now let's try editing an article. Please go to the article on "scarecrows".
Let's add the sentence "Scarecrows are often filled with straw." to the introduction of this article. Show me how you would do that. Remember to talk me through what you're seeing.
Let them preview changes
Can you make the word "straw" link to the Wikipedia article on straw?
Interested to see: if they use the toolbar buttons, or add square brackets directly. Or if they try the Cheatsheet, or look elsewhere for help.

Observations

Surprisingly nobody had any real trouble with the first part of this. Two initially clicked a section-edit link, but realised their mistake, backed out and were able to find the Edit tab. Those subjects (J, M and H) who had already made Sandbox edits were noticeably more confident in editing. A did comment negatively on the amount of "computer code" in the edit window (and this article is fairly simple compared to many others.)

Adding a link proved almost as easy. Having done much of the Tutorial earlier H was able to remember the markup for links. J and M figured it out from the other links in the wikitext and added square brackets manually. E and A found the link button in the toolbar, and used that successfully.

Notes

This part was remarkably successful, with all subjects completing a successful edit. This is at least partially due to the simplicity of the scarecrow page, with few templates and inline references. The task itself was also relatively simple, requiring only a plain unpiped link.

Task 4: Adding a footnote

Even if they didn't get anywhere with Task 2, see if they can figure this out
Now, can you add a reference for that sentence you just added. The information came from this book:
Give them slip of paper with reference details. Title: Scarecrows for Dummies, Author: A. Wurzel, Publisher: Farming Press, Year: 2012, page: 5

Observations/Notes

As the only person aware of RefTools, A went straight to the Cite option in the toolbar, and had no trouble selecting {{cite book}} and filling out the required fields. She commented on how surprisingly straightforward it was.
The other subjects struggled more, despite 'Cite' being visible none of them thought to use it (perhaps because the inconsistent terminology reference/cite/footnote made it less clear?)
J succeeded in adding a simple bare reference, though had to refer back to Help:Footnotes numerous times.
H also referred back to where she learnt about references (Wikipedia:Tutorial), struggled more, but eventually after much copy and pasting, did manage to add a bare reference.
E and M did not manage to complete this task.