Re: Advertising the Git User's Survey 2011

Hi Jakub,

Thanks for taking the time to go through my long (somewhat
unstructured) note.

Jakub Narebski writes:

On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:

quoted

Interesting statistic: 24508 people viewed it, 7821 people completed
it, but 0 people started filling out information and later decided not
to submit it. It could mean that many people clicked through and found
the survey, but probably left because it looked too long at a glance?

Without the knowledge how those two numbers are calculated we can only
speculate what do they mean.

I think that '0' in 'Incomplete' statistics is here because this survey
doesn't have compulsory questions: answering all questions are optional,
so it might mean that even if one question in the survey is answered,
then the survey is considered complete by Survs.com statistics.

I just opened up survs.com, signed up for a free account and created a
survey with two compulsory questions- I couldn't submit the results
without filling up both of them. Then I looked at the statistics and I
saw a '0' in incomplete- I'm confused. Perhaps we can email survs.com
and ask them?

I personally do not like the "wizard" formatting of surveys, i.e. 
dividing survey into page so you are not presented with very long page,
but are presented withc chunks of survey at glance.  Even if you see
how much survey did you fill in (how many pages there are in total),
and even if you can go back to previous page.

Same. I also think "wizard" is a bad idea for this- it would make it
look like some sort of online test :p

I'd prefer to create a better information about survey upfront.  We say
that all questions are optional ("Note that you may skip questions as
you like").  It is also stated that you can fill only a part of survey,
and later go back to finish it... hmmm, I wonder if those cases where
one edited his/her survey responses multiple times are counted as one
finished survey, but multiple views.

Sure. Again, we should perhaps email survs.com for such
clarifications.

We could also write how much time it takes on average to fil the survey.

Sounds good.

quoted

The average time spent on the survery is 34 minutes

It would be interesting to have more detailed statistics of time spent
on the survey that only average time, a single number.  When one is
filling open-form essay-length question, it would obviously take much
more time than for one who doesn't.

Hm, yes. I didn't think about this.

quoted

- I think we can bring that down to 10~15 minutes if we design
questions to extract more information. Also, there's little incentive
for taking the survey: while many companies actually give out
discounts/ coupons for taking surveys, the least we can do is present
real-time results in the most interesting manner possible ie. survey
takers should see the "results so far" immediately after taking the
survey; some visualizations such as pie charts?

What we can do is after finishing the survey to redirect to the
survey analysis page:

  https://www.survs.com/results/33Q0OZZE/MV653KSPI2
  http://tinyurl.com/GitSurvey2010Analysis

instead of IIRC currently used redirect to

  https://git.wiki.kernel.org/index.php/GitSurvey2010


As far as I know Survs.com doesn't provide any API for extracting data
or survey statistics required for creating such visualization.  Neither
we have a place where such app could be created, I think.

Oh, ok- we should take too much trouble though; we should keep
cost-to-benefit ratio in mind.

quoted

In questions 5, 10, 12, 13, 16, cut down out the options that have
very few respondents and let them all go into "other". It probably
doesn't actually save the survey taker any time, but I think seeing a
long page with many options can be scary.

The "5. Which Git version(s) are you using?" is not that long.  We could
create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or
even earlier, we could remove alternate implementations answers 
(git-bigfiles, JGit, other implementations), or even concatenate 'master',
'next', 'pu' into single response... but would it buy us much?

The other side of removing choices, relying instead on "other, please
specify" response is that it makes it harder to analyze results of survey:
different people use different words for the same thing (and there are
also spelling mistakes), and make results less reliable: people do not
fill "other" if there is at least partial match, or do not know how to
specify their version.

Ofcourse. We should only remove options that don't necessarily tell us
very much.

We can remove those choices in "12. What Git GUIs (graphical user
interfaces) do you use?" that got less than 1% rounded, or less than
10 responses.  On the other hand some people stated earlier that the
list of possible choices in the survey (not necessarily about this
question in specific) serve as reminder / information about possible
choices.

Hm, I hadn't thought of that. I suppose it can be used to advertise
various applications.

The other side of removing options from "13. Which git hosting site(s)
do you use for your project(s)?" is that when sending requests to 
announce the survey to those git hosting sites that are not on this
list, some of them requested to be added (which is impossible after
starting the survey; and before survey begins it is little sense to
send announcements).

I see. How are we going to tackle this in future?

Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
CipherHive) are also those that I didn't get response to request for
announcing Git User's Survey 2010...

Interesting.

We could make it more organized though, e.g. by sorting list of options
alphabetically, or something like that.

Sounds good.

quoted

1. Country of residence: we can probably make this a nice click-on-map
   interface as opposed to freeform text. It'll be more useful to us,
   and more interesting to users when we advertise the results.

It would be nice to have click-on-map (Google Maps or Bing Maps based),
something like Ohloh provides, resulting in map of survey responders
similar to the map of git users and git contributors on Ohloh

  http://www.ohloh.net/p/git/map

it isn't something that Survs.com offers currently.  I can only ask for
it to be provided...


Another solution would be to have pre-filled combo box (<select> field)
with the list of countries to choose from, with GeoIP ised to pre-select
the country.  I can generate list of all countries myself

  $ perl -MLocale::Country \
    -wle 'print join("\n", sort (all_country_names()))'

as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
to survey questions.

I suppose we could always work out a way to display the results from
the information Survs.com gives us.

quoted

2. Age: Maybe we restrict the input to 2-digit integers and draw a
   graph with all these integers to show a mean, median etc?

Restricting input doesn't give us much.

I just meant it as a sanity check in case people enter "34 years old"
and the like.

There is nice histogram of responder's age for Git User's Survey 2009

  https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F

and tabularization of responses.  We can calculate mean, median, mode
(aka modal score, i.e. most common response), perhaps after eliminating
outliers, but would it give us much information?

Nice histogram! How did we manage to do this in 2009? Did we use a
custom-made application to do the survey?

quoted

11. Just change this to an optional sometimes/ often? Why should users
    spend time clicking on "never"?

"Never" is here because you can't un-click response in given row.
I'm also not sure how not answering row is represented in the export
of survey data (which we use for further analysis).

Oh, right. I didn't think about this.

quoted

17, 18: Merge perhaps?

Those questions are split because of limitation of Survs.com; the 
"other, please specify" results in limited width _text field_, while
it is much easier to write in-depth response in large _textarea_ field
that wuestion 18 provides.

Ah, I thought so.

quoted

Ofcourse, I understand that there must be some technical constraints
due to which some things are not implementable (eg. survs doesn't
provide the feature?), but I've not taken that into consideration.

Note that as it currently stands we can use Survs.com account only for
2011 survey, provided that it is done earlier than this year (perhaps
1 June -- 31 July?), as our Premium account which we got thanks to 
generosity of Survs.com admins (after Survs.com got out of beta) will
downgrade to the Free plan (which is offersn much too low limits)
on Sep 22, 2011.

I see. Any thoughts on long-term plans? Do we pay for the premium
account or do we build a custom application?

-- Ram