1 / 16

Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys

Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys. Vanessa Torres van Grinsven & Joop Hox. Faculty of Social and Behavioural Sciences Department of Methodology and Statistics. Incentives are widely used in individual or household surveys

shauna
Download Presentation

Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys Vanessa Torres van Grinsven & Joop Hox • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  2. Incentives are widelyused in individual or householdsurveys • Proven methodtoimprove response rates in these surveys • But they are typicallynotused in official (business ) surveys • Systematic synthesis of experimental research on the effectiveness of incentives in organizational surveys is still lacking • Meta-analysis of the effect sizes of incentives in experimental research. • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Introduction Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  3. Effect size (ES) makesstatistical meta-analysis possible • It is the dependentvariable • Represents the magnitude anddirection of the relationship of interest • Must be able to calculate a standard error for that type of ES • The standard error is needed to calculate the ES weights, called inverse variance weights • Because all meta-analytic analyses are weighted taking into account sample size. • More information: Lipsey & Wilson (2001) • In this case the effect size is the difference in response ratebetween the treatment groupsand the corresponding control groups • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Short introduction on statistical meta-analysis Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  4. Search strategy: • Search made in numerous databases: e.g. Web of Science, WorldCat, EBSCO, Picarta, PsycInfo, SIAM, ESOMAR (the research papers database), AAPOR (Survey Practice database). • Search words: incentive, survey, experiment • Eligibility criteria: • Experiment with split ballot • Appropiate control group • RR1 reported or calculable • Sample sizeandgroupsizes are reported • Organizational survey • Incentives • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Data collection Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  5. Organizational survey defined as survey on: • A commercial population: “Commercial populations are defined as those composed of commercial, industrial, administrative and/or business respondents who receive a questionnaire at their place of employment” Pressley & Tullar, 1977, p. 108. • Non-profit organizations • Other professionals whoreceive the questionnaire at theirplace of employmentandabout job-related issues. • Incentive: a prepaid or promised monetary or non-monetary incentive. • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  6. Studycharacteristicswerecodedtounderstandifthey had an effect on the effect size • Codingschemebased on open codingandcodingschedulesby De Leeuw et al. (2007) andHox & De Leeuw (1994) • Also effect size information was coded (the response rates) • Codedstudy features were: year of publication, country, research organization, sample frame, population type, survey topic, survey method, data type, industry, respondent type, voluntary or mandatory survey, type of incentive, andvalue of the incentive. • Additionally(incentive): prepaid vspromised, monetaryvsnonmonetary, type of monetary incentive, type of nonmonetary incentive. • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Coding of studycharacteristics Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  7. 34 studies with a total of 68 experimentalconditions. • Combined sample size of 343.500 respondents including the control groups. • The response rates are expressed as proportions, which are then transformed into Z-values using the standard normal distribution. • The standardized effect size d is then given by: • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Data analysis (1) Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  8. And the standard error of d is given by: • Because a larger standard error corresponds to a less precise effect size value, the optimal weights are computed as the inverse of the squared standard error value – hence inverse variance weight (Hedges 1982; Hedges & Olkin, 1985): • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Data analysis (2) Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  9. Inverse varianceweighted meta-analysis techniqueswith a random effects model withrestricted maximum likelihoodestimation (REML) • The distribution of effect sizes was heterogeneous • Meta-regressionand the meta-analyticanalogue of ANOVA wereusedfor the moderator analysis • SPSS Wilson’s meta-analysis macros (http://mason.gmu.edu/~dwilsonb/ma.html ) • More information Lipsey & Wilson (2001) Data analysis (3) Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  10. A relevant issur in meta-analysis is publication bias • It refersto the possibilitythat relevant studies mightnot have been published, but disappeared in the file-drawer • Publication bias was assessedbyinspecting a funnel plot andcalculating the fail-safe N. • Fail-safeN: the significant effect of the incentive on the response rate would be swiped away if we assume the existence of 426 unpublished studies all reporting no effect highly implausible • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Publication bias Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  11. Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Publicationbias:Funnel plot of the correlation between effect size and sample size Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  12. Sending or promisingan incentive does have a small but significant effect on the response rate • Mean effect sized = .25 (p<.0001) • In terms of Cohen’s (1988) effect sizedefinitionsthis is a small effect • Of all the codedstudy features onlyresearch organizationhad a significant effect on the effect size (p=.04) • Commercial organizations: d = .36 • Universities: d = .26 • Governmentalinstitutions: d = .10 • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Results Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  13. Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Value of the incentive: Effect size (d) related to the amount of the monetary incentive Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  14. The largest effect canbeexpected in commercial surveys; the smallest in surveysbygovernment-relatedinstitutions, likeNSOs. • If we include the response rate as a predictor, the significant differencesbetweenorganization types disappearscompletely; while the regressioncoefficientfor the response rate is negativeandhighly significant • Thus: research organizations that already are obtaining high response rates, find it difficult to increase them further • We concludethat incentives canbeused as a general tool toincrease response in organizationalsurveyswith different survey features • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Results Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  15. Whatseems important thus is not the factualvalue(as in aneconomic exchange) • But the psychologicalvalue: • Reciprocitytheory(Gouldner, 1960): • a symbolic gift thatneedstobereciprocated: • A recipientmay feel obligatedtoreciprocatebycompleting the questionnaire • Cost/benefit theory(e.g. Singer, 2011): • The incentive is seen as a symbolic benefit that counteracts the perceivedcosts of completingandreturning the questionnaire • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics Discussion Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

  16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum • De Leeuw, E., Callegaro, M., Hox, J., Korendijk, E. & Lensveldt-Mulders G. (2007). The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis. Public Opinion Quarterly, 71(3), 413-443. • Gouldner, A. W. (1960). The norm of reciprocity: A preliminary statement. American Sociological Review, 25, 161-178. • Hedges, L.V. (1982). Estimating effect size form a series of independent experiments. Psychological Bulletin, 92, 490-499. • Hedges, L.V. and Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, Florida: Academic Press. • Hox, J.J., & De Leeuw, E.D. (1994). A comparison of nonresponse in mail, telephone, and face-to-face surveys. Applying multilevel modeling to meta-analysis. Quality and Quantity, 28, 329-344. • Lipsey, M.W. & Wilson, D.B. (2001). Practical Meta-Analysis. Sage Publications. • Pressley, M.M. & Tullar, W.L. (1977). A Factor Interactive Investigation of Mail Survey Response Rates from a Commercial Population. Journal of Marketing Research, 14(1), 108-111. • Singer, E. (2011). Toward a Benefit-Cost Theory of Survey Participation: Evidence, Further Tests, and Implications. Journal of Official Statistics, 27, 379-392. • Faculty of SocialandBehavioural Sciences • Department of MethodologyandStatistics References Methodology and Statistics | Meta-Analysis of the Effectiveness of Incentives in Organizational Surveys |

More Related