Wikipedia talk:Wikipedia Signpost/2009-09-07/In the news
Re the survey, The respondents were self-selected, and there were unexplained anomalies in response rate, including a dramatic over-representation of Russian Wikipedia users (whose responses were not included in the gender statistics and other survey results and analysis). -- as far as I recall from the Wikimania session, the Russian responses were initually excluded (as reported in the WMF blog), but after being scrutinised they were folded back in (as reported in the Wikimania session). --pfctdayelise (talk) 00:54, 8 September 2009 (UTC)
- Thanks for the info. Unfortunately, the survey results session was one of the few sessions that apparently didn't get recorded (or least, there's no video available that I know of). I'm curious as to what the explanation was for the high response rate (and curious if there was much pushback from attendees in terms of the methodology and the reliability of the resulting stats).--ragesoss (talk) 01:44, 8 September 2009 (UTC)
- It lead to an interesting discussion in Buenos Aires, and some flaws in the methodology emerged. The study was not preceded by sufficient focus group work or other testing to get the questions right so there were some faults in the questionnaire design, in particular the age question restricted the survey to 18-85 year olds (which could partially explain the high drop out rate - I think it was 40% who opened the survey but didn't complete it). Losing the under 18s will obviously skew the academic qualification question and probably others. Also the question as to why people edit Wikipedia didn't include the answer "for fun", and someone there from a similar mass volunteer project suggested that based on their research that would have been the dominant answer if our volunteer base was similar to theirs. The Russian data was tested and eventually incorporated because in many ways it was similar to that from other projects. I think it was this research project where they didn't separate out arts and culture from popular culture and were surprised at how much of the Wiki activity was arts and culture, whilst the feedback from the room was that excluding the under 18s had probably resulted in an under representation of the amount of pop culture editing on the pedias. They didn't seem to have done a weighting or sampling exercise to turn the survey into a representative one, though possibly this was because they wouldn't have known what to weight against... And their way to deal with vandalism was to exclude the under 18s rather than to include logic traps to identify and screen out vandal responses. But still 175,000 responses made for some interesting slides. ϢereSpielChequers 06:18, 11 September 2009 (UTC)
- Hi WereSpielChequers, I read these recollections with great interest while compiling this blog post. The bit about the age restriction is contradicted by the final report though, which said that “25% are younger than 18 years old”).
- @Ragesoss: A video exists, I have embedded it at the new documentation page I just started for this survey: m:Research:UNU-MERIT Wikipedia survey.
- Regards, Tbayer (WMF) (talk) 09:02, 1 May 2015 (UTC)
- @Tbayer (WMF): I can't find the original survey, just the results, and six years on I am just relying on memory and what I wrote at the time. But it is entirely possible there was both the up front question that I remember restricting it to 18+ and subsequently an age question that allowed for under 18s, if so it would at least partially account for the 40% drop out figure. Especially as the peak age group was 18 (The most frequent age that can be observed within the respondents is 18 years). Whilst their admission that they discarded age outliers should be remembered when we say that the respondents ranged from 10-85 - if we repeated it now, and I hope we do, I would hope we would not assume that no-one over 85 edits. ϢereSpielChequers 09:23, 1 May 2015 (UTC)
- It lead to an interesting discussion in Buenos Aires, and some flaws in the methodology emerged. The study was not preceded by sufficient focus group work or other testing to get the questions right so there were some faults in the questionnaire design, in particular the age question restricted the survey to 18-85 year olds (which could partially explain the high drop out rate - I think it was 40% who opened the survey but didn't complete it). Losing the under 18s will obviously skew the academic qualification question and probably others. Also the question as to why people edit Wikipedia didn't include the answer "for fun", and someone there from a similar mass volunteer project suggested that based on their research that would have been the dominant answer if our volunteer base was similar to theirs. The Russian data was tested and eventually incorporated because in many ways it was similar to that from other projects. I think it was this research project where they didn't separate out arts and culture from popular culture and were surprised at how much of the Wiki activity was arts and culture, whilst the feedback from the room was that excluding the under 18s had probably resulted in an under representation of the amount of pop culture editing on the pedias. They didn't seem to have done a weighting or sampling exercise to turn the survey into a representative one, though possibly this was because they wouldn't have known what to weight against... And their way to deal with vandalism was to exclude the under 18s rather than to include logic traps to identify and screen out vandal responses. But still 175,000 responses made for some interesting slides. ϢereSpielChequers 06:18, 11 September 2009 (UTC)