The missing 2012 editor survey

If one of the stated goals of the WMF is to improve the gender gap, the place to start is with the metrics: What is the percentage of male and female contributors? How do you craft programs to address this issue? And most of all, how do you measure the effectiveness of the programs that are in place?

Where are the survey results?

The survey was conducted in April of 2012, but the results have still not been posted, in spite of many requests.  The last request was in March 2013, and at that time the WMF staff made assurances, but there has still been no follow-through.  The results are now being requested again on the survey talk page, along with some indication of a timetable for posting the raw data.

The earliest survey I can find is the UN University in Maastrict (UNU-MERIT) 2008 survey.  The results were posted on 24 March 2010, and are are available here.  There is some indication that the WMF intended to carry out this survey every 6 months. There are data sets of other past surveys posted here.

The 2012 survey

Editor_surveys_(Wikimania_Hong_Kong,_August_9,_2013).pdf slide 20Some results from the survey are available.  John Vandenburg posted some slides from the Wikimania 2013 presentation to Facebook by in August 2013. Other data is available on a collection of slides posted to Commons, which apparently includes the slide set used at Wikimania Hong Kong, August 9, 2013, as well as a subset of the Wikimania slides dealing with WMF satisfaction presented to a WMF metrics meeting 2013-09-05.

There is also a video of the Wikamania 2013 presentation on YouTube.

In the question session at the end, there was a question about slide 21, that shows 18 percent of respondents find female editors to be an inappropriate priority for the Foundation. One of the participants asked the breakdown of the respondents to that question by gender.

Editor_surveys_(Wikimania_Hong_Kong,_August_9,_2013).pdf slide 21

Who wrote the survey?

Unlike the survey conducted by UN University in Maastrict (UNU-MERIT) in 2008, the 2013 survey was written by WMF staff and contractors:

Editor_surveys_(Wikimania_Hong_Kong,_August_9,_2013).pdf slide 25
Mani Pande https://www.linkedin.com/in/manipande
Mani Pande

Ayush Khanna (intern? recent grad) http://ayushkhanna.com/
Ayush Khanna

Elizabeth MH (Consultant to the LCA Team) http://blog.wikimedia.org/2012/09/17/wikimedia-foundation-report-august-2012/ (legal and community advocacy)

Philippe Beaudette http://wikimediafoundation.org/wiki/User:Philippe_%28WMF%29
800px-Making-Wikipedia-Better-Photos-Florin-Wikimania-2012-26 Philippe

There has been some feedback about the survey form on the research talk page.

Cooking the books? Statistical manipulation with “propensity scoring”

After the 2008 UN University in Maastrict (UNU-MERIT) survey was completed, the WMF used a statistical manipulation called “propensity scoring” to revise the results upward, apparently without consulting UNU-MERIT, resulting in higher numbers for female participation. This was a topic of the July 2013 Wikimedia Research Newsletter and was also described on Benjamin Mako Hill’s blog. There was also an article titled “The Wikipedia Gender Gap Revisited: Characterizing Survey Response Bias with Propensity Score Estimation“, by Benjamin Mako Hill and Aaron Shaw. Hill is a member of the Wikimedia Advisory Board.

Advertisements

One thought on “The missing 2012 editor survey

  1. from http://thread.gmane.org/gmane.org.wikimedia.gendergap/5338/focus=5341 :

    From: Risker <risker.wp gmail.com>
    Subject: Re: Fwd: Update on 2012 editor survey results
    Newsgroups: gmane.org.wikimedia.gendergap
    Date: 2015-01-15 04:30:05 GMT (23 weeks, 5 days, 16 hours and 55 minutes ago)

    Well, to be perfectly honest…

    Think back to that survey, if you can. It was written entirely in-house, by people with no particular skill in survey development, no genuine experience in writing non-leading questions, etc; it was amongst the most unscientific studies I’ve participated in while wearing my Wikipedia hat. It went on and on about chapters, which was a totally meaningless question for over 90% of editors; I remember that much because it ticked me off. There was some pretty good indication that it was enwiki-centric and what wasn’t enwiki-centric was Wikipedia-centric. I don’t know that any result from that survey will mean much from a research point of view. The fact that the WMF has never bothered to publish the data suggests to me that even they realised how flawed the data collection process was.

    And for the record, I ticked that I didn’t want to say my gender.

    Risker/Anne

    You go, Risker!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s