Transparency in Public opinion Research

1. Transparency in Public opinion Research Timothy Johnson Survey Research Laboratory University of Illinois at Chicago

2. “Men whose research is based on shared paradigms are committed to the same rules and standards for scientific practice.” • -Thomas S. Kuhn, The Structure of • Scientific Revolution. 1962

3. Is public opinion research in a state of crisis?

4. Is public opinion research in a state of crisis?

5. WAPOR Freedom to Publish Opinion Poll Results: A Worldwide Update 2012 • 85 nations examined • Key findings • Black-out periods for pre-election polls: 46% • Any restrictions on conducting exit polls: 59% • Any restrictions on publishing exit polls: 62% • Any restrictions on topics covered by polls: 14%

6. Some Rationales for Polling Restrictions • Polls trivialize elections by turning them into a horse race • Polls displace serious coverage of candidates & issues • Polls encourage negative media advertising campaigns • Polls encourage complacency among frontrunners and defeatism among trailing candidates • There is a conflict of interest when media conduct polls to generate news • Some politicians follow the polls rather than their convictions • Polls can be manipulated to influence and mislead

11. Types of Regulation • Government regulation • Covered by WAPOR report • Self regulation • Voluntary adherence to professional standards

12. Types of Regulation • Government regulation • Covered by WAPOR report • Self regulation • Voluntary adherence to professional standards “if national and international professional associations advance scientific and quality standards for conducting surveys and promote disclosure standards for survey methods, they will undermine the case for banning, censoring, or restricting polls.” -Tom Smith (2004) International Journal of Public Opinion Research

13. “Good professional practice imposes the obligation upon all public opinion and survey researchers to disclose sufficient information about how the research was conducted to allow for independent review and verification of research claims.” • AAPOR Code of Professional Ethics and Practices (Revised 11/30/15)

15. Why a Transparency Initiative? There is not enough transparency in the Social, Behavioral & Marketing Sciences. • Premise 1: Too often, there is inadequate transparency of research and statistical methods when the results of social and behavioral science research are disseminated. • Premise 2: This is a serious detriment to progress in these scientific domains. • Premise 3: Lack of transparency contributes to public disillusionment.

16. Openness is fundamental to scientific inquiry and credibility. • AAPOR’s Transparency Initiative (TI) is designed to encourage openness in survey research so the field can advance. • TI is envisioned to enhance the credibility of our profession by encouraging disclosure of methodological information and educating the public on how to use it.

17. History of the Transparency Initiative • 2010 AAPOR Conference: AAPOR President Peter Miller announces TI. • 2010-11: Steering Committees worked to conceptualize and organize TI. First pilot test. • 2011: WAPOR endorsement of TI. • 2012-13: Multi-year TI Coordinating Committee appointed and charged with conducting second Pilot Test. • 2013: Second Pilot Test conducted. • 2014: TI officially launched; AAPOR journals adopt TI standards. • 2015: US Federal Committee on Statistical Methodology acknowledgement of TI. • 2016: total of 80 members accepted as of 12/1/16.

18. Transparency Initiative Members • ABC News • Abt-SRBI • American Association for Retired Persons • American Institutes for Research • American Legacy Foundation • Buendía y Laredo S.C. • California Health Interview Survey • California State Fullerton • Castleton Polling Institute • Center for Effective Philanthropy • Chicago Council on Global Affairs • Cornell University • D3 Systems • Decision Information Resources, Inc. • EdChoice • Elon University Poll • Emerson College Polling Society • Fairleigh Dickinson University • Gallup Organization • Gomez Research, Inc. • Goucher College • High Point University • Huffington Post • ICF International • iMediaEthics • Indiana University Center • InterMedia Survey Institute • Ipsos Public Affairs U.S. • JD Granz Research, Inc. • Kaiser Family Foundation • Langer Research Associates • Latin American Public Opinion Project at Vanderbilt University • Loyola Marymount University • Marist Poll • Marquette University • Mathematica Policy Institute • Michigan State University • Monmouth University • Muhlenberg College

19. Transparency Initiative Members, continued • National Research Center, Inc. • NORC at the University of Chicago • Pew Research Center • PolicyInteractive • Public Agenda • Public Policy Institute of California • Public Religion Research Institute • Quinnipiac University • Research Services at the Presbyterian Church • Roanoke College • RTI International • Rutgers University Eagleton Center • Sacramento State University • Siena College • SoonerPoll • Southern Illinois University • SSRS • Temple University • University of Arkansas at Little Rock • University of Chicago • University of Florida • University of Illinois at Chicago • University of Illinois at Springfield • University of Kentucky • University of Massachusetts Boston • University of Michigan Ctr for Political Studies • University of Nebraska • University of New Hampshire • University of North Florida • University of Northern Iowa • University of Pittsburgh • University of South Carolina • University of Virginia • University of West Georgia • University of Wisconsin • Urban Institute • Vanderbilt University • Washington Post • Westat • Western New England Univ • Winthrop University

20. TI Members by Sector(as of December 1, 2016)

21. Revised AAPOR CodeDisclosure Items for Surveys Report Immediately • Who sponsored & conducted • Exact question wording • Definition of population & geographic location • Dates of data collection • Sample frame description & any non-coverage • Name of sample supplier • [if panel]: recruitment methods • Detailed sample design • Method(s)/mode(s)/language(s) • [if probability]: sample size & error & any design effects adjustments • [if non-prob]: sample size & precision ONLY if detailed description of underlying model • Weighting & sources of parameters • Who contact for more information Within 30 Days of request • Panel management procedures • Interviewer training, supervision & monitoring • Screening procedures • Relevant stimuli (show cards) • Strategies used to gain cooperation • Data quality procedures • Sample dispositions • [if reported]: AAPOR response and cooperation rates • Unweighted sample sizes for subgroup estimates • Specifications for replication of indices or statistical models

22. Revised AAPOR Code Disclosure Items for Qualitative Research* Include in any report or make available upon report release • Who sponsored & conducted research • Definition of population & geographic location • Instrumentation used & languages • Description of relevant stimuli (eg visual/sensory exhibits/show cards) • Dates of data collection & physical location • Subject eligibility • Number of research subjects, by data collection strategy • Methods of interviewer/coder training, supervision and monitoring • Duration of research participation • Any compensation or incentives provided to research subjects • Whether or not data collection included audio or video recordings used *includes “focus groups, in-depth interviews, case studies, narrative research, and ethnography”

23. Revised AAPOR Code Disclosure Items for Content Analyses* Include in any report or make available upon report release • Who sponsored & conducted research • How content was collected or obtained • Dates analyzed, languages, inclusion decision rules • Discussion of threats to validity/quality & steps to address • How analysis was conducted • Coding process & coding schemes used • Inter-coder reliability & number of coders • Unit of analysis • [if relevant]: weight calculations & variables used • Other relevant information if surveys or qualitative data used *Defined as the “systematic analysis of text, images or other content”

25. Revised AAPOR Code Section III.E Reflecting the fundamental goals of transparency and replicability, AAPOR members share the expectation that access to datasets and related documentation will be provided to allow for independent review and verification of research claims upon request. Datasets may be held without release for a period of up to one year after findings are publicly released to allow full opportunity for primary analysis. In order to protect the privacy of individual respondents, such datasets must be de-identified to remove variables that can reasonably be expected to identify a respondent. Those who commission publicly disseminated research have an obligation to disclose the rationale for why eventual public release or access to the datasets is not possible, if that is the case.

26. Future Plans • Develop new professional and public educational programs centering on transparency • Identify new strategies to encourage “spirit” of transparency • Long term goals: • Collaborate with other professional associations to advocate for transparency of research methods • Bring sunlight to non-traditional research methodologies • Increase: • Public knowledge and awareness of research transparency • Public demand for research transparency David Lambert

27. 2015-2016 Transparency Initiative Coordinating Committee • James Dayton • ICF International • Tim Johnson • University of Illinois at Chicago • Scott Keeter • Pew Research Center • Ashley Kirzinger* • Kaiser Family Foundation • Ronald Langley • University of Kentucky • Mary Losch • University of Northern Iowa • Peter Miller • Census Bureau • Tom Miller • National Research Center, Inc. • Rich Morin • Pew Research Center • Andy Peytchev • University of Michigan • Dianne Ruchinski • OS/OASH • Mandy Sha • Research Triangle Institute • Trevor Tompson • Associated Press-NORC Center for Public Affairs Research • Tim Triplett* • Urban Institute David Lambert *Committee Co-Chair

28. Questions? timj@uic.edu