1 / 19

M oderators of panel conditioning in sensitive questions. A meta-analysis.

M oderators of panel conditioning in sensitive questions. A meta-analysis. Tanja Burgard, ZPID Trier Nadine Kasten, University of Trier Michael Bosnjak, ZPID Trier. Talk at the Conference of the European Survey Research Association (ESRA) 18.07.2019, Zagreb ( Croatia ).

infinity
Download Presentation

M oderators of panel conditioning in sensitive questions. A meta-analysis.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Moderators of panel conditioning in sensitive questions. A meta-analysis. Tanja Burgard, ZPID TrierNadine Kasten, University of TrierMichael Bosnjak, ZPID Trier Talk at the Conference ofthe European Survey Research Association (ESRA) 18.07.2019, Zagreb (Croatia)

  2. Definition andrelevanceofpanelconditioning An example: Veroff, Hatchett and Douvan (1992) randomly assigned newlywed couples to one of two groups: one that participated in frequent and intensive interviews (the study group) about marital satisfaction [..] and another that participated in minimal and infrequent interviews […]. The authors concluded that ‘‘[b]y the fourth year . . . The marriages of the study group couples appeared to be better adjusted on several dimensions of marital quality’’ (p. 315). • Panels arenecessarytoanswer longitudinal researchquestions, but dangerstovalidity: panelmortalityandpanelconditioning • Panel Conditioning = Learning effect in panelstudies • Problem: Due totheconditioningofrespondents in formersurveywaves, theyarenolongerrepresentativefor non-respondents in laterwaves. • Examples: Avoidanceof follow-upquestions, cristallizingofattitudes, increasedattentionforsurveytopics, knowledgechanges Warren, Halpern-Manners (2012): Panel Conditioning in Longitudinal Social Science Surveys. SociologicalMethodsand Research 41(4): 491-534.

  3. sensitivityanditsrelevanceforpanelconditioning • Threecharacteristicsof sensitive questions (Tourangeau & al., 2000): • Answercallsforsociallyunacceptedanswer • Intrusive and private, taboo in everydayconversation (religion, income, sexuality) • Concernfordataprivacy, datasecurityanduseofdata Twopossibleeffectsofpanelconditioning in caseof sensitive questions: • Desirable: More trust in surveysituation→  More honest answers, lesssocialdesirabilityeffectsforattitudequestions • Not desirable: Consistencyeffect on behaviororself-reporting→ Adjustmentofactualorreportedbehaviortogreaterconformity in caseof deviant behavior

  4. Measurement ofpanelconditioning Time 1 Time 2 Time 3 Group 1 T T • Between-subjects design: Comparisonofparametersofexperiencedandfreshparticipants C Group 2 C Group 3

  5. Hypotheses • H1: The answersofexperiencedrespondentsdifferfromtheanswersoffreshrespondents in caseof sensitive questions. • H2: Socialdesirabilityeffects in sensitive questionsdifferdepending on the type ofquestion. • H2.1: Experiencedrespondentsanswerlesssociallydesirable in caseofattitudequestions. • H2.2: Experiencedrespondentsanswermoresociallydesirable in caseofbehaviorquestions. • Dosageeffects • H3: The moreoftenthetreatmentgroup was interviewed, thegreaterthedifferencebetweengroups. • H4: The greaterthe time-lag betweenwaves, thesmallerthedifferencebetweengroups. • H5: The kindofsensitivityinteractswiththe type ofquestion • H5.1: Questionsthatcallforsocialdesirableanswersincrease PC effects in caseofbehaviorquestions. • H5.2: Intrusive and private questionsdecrease PC effects in caseofattitudequestions.

  6. Information searchandselection • First literaturesearch: • CLICsearch(broadsearchinterfacecontainingforexamplePsycInfo, PsycArticles, PubMed, Sociological Abstracts) • Search terms: „panelconditioning“, „surveyconditioning“, „time in sample“, „rotationgroupbias“ and 10 morerelatedsynonyms • Eligibilitycriteria: • (Quasi-) experiments • Population: Experiencedandfreshpanelrespondents (treatmentandcontrolgroup) • Treatment: Fomerconditioningby sensitive surveyquestion • Time ofcomparison: Bothgroupsareaskedforthe same, sensitive question • Outcome: Reportedbehaviororattitudesofbothgroups SMD • Forward andbackwardsearchwithrecordsfromthefirstsearchmeetingeligibilitycriteria

  7. PRISMA Flow Chart

  8. Data collectionandoutcome • Coding • Information on thereport: Author, Year ofpublication, funding • Intervention: Dataset, surveymode, yearofcomparison, countryofconduction, incentives, Type ofquestion, topic, conditioningfrequency, intervalbetweenwaves • Results: Outcomes ofbothgroups, Odds Ratios, teststatistics • Effectsizemeasure: Standardizedmeandifferences • SMD > 0: Experiencedpanelistsrespondlesssociallydesirable • SMD < 0: Experiencedpanelistsrespondmoresociallydesirable

  9. Analysis method • Toaccountforthehierarchicaldatastructure, a three-level meta-analysis isused • Source: Harrer, M. & Ebert, D. D. (2018). Doing Meta-Analysis in R: A practical Guide. PROTECT Lab Friedrich-Alexander University Erlangen-Nuremberg. https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/ • R packageused: metafor 2.0-0

  10. Overall effect (H1) andvariancedistribution • k = 154 effectsizes, x = 85 samples, n = 19 reports • Meaneffectofpanelconditioning: -0,028*** [-0,042; -0,013] •  Overall, experiencedpanelistsrespondmoresociallydesirablethanfreshpanelists  H1 • Distribution ofheterogeneity: • Sampling variance: 5,26 % • Withinstudies: 80,40 % • Betweenstudies: 14,33 % True heterogeneity, thatmaybeexplainedwithmoderators

  11. Moderating effectof Type ofoutcome (H2) • H2.1: Experiencedrespondentsanswerlesssociallydesirable in caseofattitudequestions. • H2.2: Experiencedrespondentsanswermoresociallydesirable in caseofbehaviorquestions. • 13,6 % ofthevarianceofSMD‘swithinstudiesand 37,5 % ofthevariancebetweenstudiesisexplainedbythe type ofoutcome

  12. Dosageeffects (H3 / H4) • Gesamt

  13. kindofsensitivityand type ofquestion (H5) • H5.1: In caseofquestionsthatcallforsocialdesirableanswers, PC effectsforbehaviorquestionsarestronger. • H5.2: In caseof intrusive and private questions, PC effectsforattitudequestionsaresmaller.

  14. Summarizedfindings

  15. Conclusions • As expected, PC effectsdiffersignificantlybetweenattitudeandbehaviorquestions. • Noevidencefor PC effects in attitudequestions • Experiencedrespondentsreportmoresociallydesirablebehavior • Moderators ofbehaviorquestions • Frequencyslightlyincreasesthe PC effect • Contradictorytoexpectations, time lageincreases PC effect plausible, asbehaviorislearnedover time? • Questionscallingforsocialdesirableanswersincrease PC effects in behaviorquestions • What‘snext? • PC effectsforotherkindofoutcomes (demographics, wellbeing, knowledge) • Targeted experimental studiesevaluatingtheeffectsoffrequencyandtiming

  16. Thankyouforyourattention! Questions?!?

  17. Manifestationsofpanelconditioning Panel conditioning in thecontextoftheansweringprocess in surveys(Tourangeauet al. 2000): • Stage 1: Comprehensionofthequestion [-] Change in attitudesorbehavior due toreflection / increasedattention(Sturgis et al 2009: Cognitivestimulusmodel) [+] Less„don‘tknow“-answers • Stage 2: Information retrieval [+] More reliableanswers due tobetteraccessibityof relevant information(Bergmann, Bath 2017) • Stage 3: Assessment ofavailableinformation [-] Freezingofattitudestoappearconsistently(Waterton, Lievesley 1989) • Stage 4: Reporting / Selectionofadequateanswer[+] Reductionofsocialdesirabilitybias more honest answering(Waterton, Lievesley 1989)[-] Reductionofthecognitiveburdenofthesurveybystrategicanswering/ satisficing(Krosnick 1991): • Negative answeringoffilterquestionstoavoid follow-upquestions • Selectionofacceptableanswerswithoutprocessingthecontent Dynamics and moderators of panel conditioning. A meta-analysis

  18. Relevanceandproblemsof Panels • Demand ofthesciencecouncil: Infrastructure in psychologyfor longitudinal datacollection Bruder et. al (2014): Nationallyfounded online lab • Existing panel-infrastructures: NEPS; GESIS Panel, LISS Panel (Blom et al 2016)  ZPID: Online-Lab • Relevanceofpanelinfrastructures • Improvementofresearchpossibilities (availabledatabasefor longitudinal researchquestions) • Provision byinfrastructuremoreefficient: • Recruitmentand care forpoolofparticipants • Methodologicalandtechnicalexpertise • Dangersofpanels: panelmortalityandpanelconditioning

  19. Rationale fortheexpected time effectofpanelconditioningeffects • Pluralism / lessbindingnessofsocialnorms • Socialdesirabilitylessimportantfornewrespondents, too • Information overloadandscarcityofattention • Cognitivestimulus due tosurveyparticipationlesspronounced • Information ofprevioussurveysarelessaccessible due toamountofinformation • Increase in surveysandscientificstudies • More familiaritywiththerulesofsurveys • Satisficingandstrategicansweringismore probable withnewrespondents, too • General tendency: Decreaseofpanelconditioning, becauserespondentsarelessaffectedbythesurveyparticipationandthus, differencesbetweennewandexperiencedrespondentstendtolevel out Dynamics and moderators of panel conditioning. A meta-analysis

More Related