Andrey Bykov and Inna F. Deviatko National Research University Higher School of Economics Moscow

1. ESA RN21 / EQMC Conference, Mannheim, October 24-25, 2014 It Takes All Sorts to Make a (Survey) World: Differential Effects of Survey Mode for the Factorial Vignettes and a Self-Report Personality Inventory AndreyBykov and Inna F. Deviatko National Research University Higher School of Economics Moscow

2. Methods of social science research: online vs. “traditional” (offline) • Rapid increasing of Global Internet access – potentially representative samples • Convenient for both participants and researchers • Administering research online could be relatively cheap and fast • However: the problem of equivalence!

3. Equivalence of online and offline modes of administration • Numerous studies concerning surveys and psychological self-report inventories (Worthy and Mayclin, 2013; Wang et al., 2013; Gravlee et al., 2013; Kleck and Roberts, 2012, and others) • Generally, the modes are found to be equivalent, although, usually with certain peculiarities • To our knowledge, there are very few published studies (Nekrasov, 2011) on the modes’ equivalence of factorial vignettes (Rossi and Anderson, 1982)and no such studies concerning the Self-Report Altruism Scale (Rushton et al.,1981) • In our experiment, we examined the mode equivalence of the altruism-related factorial vignettes and the SRA scale

4. Factorial Vignettes • Vignettes are short, coherent, usually fictional, descriptions of situations or persons which are presented to the subjects as a kind of experimental stimuli and followed by questions or rating scales • Factorial vignettes assume a model consisting of several factors, each with several levels, with vignettes being unique combinations of these levels phenomena (Rossi and Anderson, 1982) • There can be different experimental designs (random, quota, full-factorial) • FV are very convenient for administering online

5. The Self-Report Altruism Scale • Introduced by Rushton et al. (1981) as a tool for measuring individual proneness to altruistic behavior • A relatively simple Likert-type scale which consists of 20 items describing different altruistic actions, such as helping strangers, donating to charity, et cet. • The participants are instructed to check the frequency with which they have engaged in these actions on a 5-point scale ranging from ‘never’ to ‘very often’

6. Method: participants and recruiting • Two samples: offline (PP – paper and pencil) (73 students of HSE) and online (WB – Web based) (93 students of HSE and other Russian universities) • PP participants were recruited in classroom. No reward was given for participation • WB participants were recruited partly in classroom and partly online (via invitation after completing another study or an advertisement in a popular social network). No reward for participation.

7. Method: research design • The questionnaire we used in our experiment (which was a part of a larger study of altruistic actions) consisted of factorial vignettes and the SRA scale, also including a block of questions about the participants’ evaluation of the questionnaire’s subjective difficulty • The vignettes described two persons, X and Y, in a situation where X was in need (actually, we used different personal names instead of X and Y). After reading each vignette, participants were asked to assess whether, in their opinion, Y should give X the required donation using 11-point scale ranging from 0 (absolutely should not) to 100 (absolutely should) • Full-factorial within-subject design. The experimental plan included 4 factors with 2 levels each: 1) relatedness between the two persons (relative vs. non-relative), 2) history of their relationships (X often helped Y in the past vs. X seldom helped Y in the past), 3) the possibility to meet in the future (moderate vs. high), and 4) the amount of required help (low vs. high). Therefore, every participant had to rate all the 16 (2*2*2*2) vignettes • The vignettes and the SRA scale were presented to the subjects in a counterbalanced order • WB and PP questionnaires were maximally identical in terms of design (2 vignettes per page, 10 SRA scale items per page).

8. Analysis and results • Completion time (in seconds): ANOVA, F=11.17; p≤0.001, eta squared = 0.064.

9. Analysis and results • No significant differences between the modes for the FV

10. Analysis and results • However, our analysis also showed a very slight, but significant difference between the modes for the total SRA scores. In the PP mode, mean SRA score was 47.92 (σ=8.6), while 51.22 (σ=8.9) in the WB mode. ANOVA confirmed statistical significance at p=0.018 with F=5.75 (eta squared =0.034). • The SRA scale’s internal consistency (reliability) was almost equally high in both conditions (Cronbach’s alpha was 0.81 for offline and 0.80 for online).

11. Analysis and results (the SRA scale’s factorization) WB mode PP mode Eigenvalues of the factors in principal component analysis of the SRA scale (varimax rotation)

12. Analysis and results (the SRA scale’s factorization) • According to both Kaiser Eigenvalue and the Cattell’sscree plot criteria, we defined the number of factors for further CFA investigation: 6 factors for PP mode and 5 factors for WB mode. These factors respectively explained 60.6% of total variance for offline mode SRA data and 51.6 % - for online mode • Individual items loadings in PP and WB modes were found to be partially coinciding for the three first factors and rather different for the rest.

13. Analysis and results (subjective comfort, FV) (Chi Squared=18, df=4, p<0.001).

14. Analysis and results (subjective comfort, SRA scale)

15. Discussion 1. First, our study showed that participants spent more time completing the web-based questionnaire compared to the paper-and-pencil one Possible explanation: other activities while filling in the questionnaire (drinking coffee, surfing the Web…) 2. Second, we found no significant differences in the assessments of factorial vignettes’ between WB and PP modes Possible explanation: there is no survey mode effect for the factorial vignettes 3. Third, our analysis showed a slight but significant effect the survey mode has on the Self-Report Altruism Scale total scores: these scores were a little higher for the participants who completed the online questionnaire Possible explanation: self-selection of the WB mode participants

16. Discussion (continued) 4. Fourth, we conducted a factor analysis of the Self-Report Altruism Scale separately for both PP and WB modes of administration: the factor structure was found rather different but partially coinciding Possible explanation: different recruiting procedures and unequal group sizes 5. Finally, participants in our WB mode study reported that working with factorial vignettes was for them rather difficult, while those who used a PP questionnaire did not think so. We did not find such differences for the SRA scale Possible explanation: rating of a number of complex factorial vignettes contradicts people’s general expectations of the Internet activities

17. Thank you for your attention!