1 / 48

A Comparison of Survey Reports: Standard Questionnaire vs. Event History Calendar

This study compares survey reports obtained through a standard questionnaire and an event history calendar. It evaluates the priming effect and explores the potential benefits of the event history calendar method. The study aims to improve the design, data quality, and cost-effectiveness of the survey.

fleos
Download Presentation

A Comparison of Survey Reports: Standard Questionnaire vs. Event History Calendar

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar* Priming Evaluation * • Jeff Moore, Jason Fields, Gary Benedetto, • Martha Stinson, Anna Chan, and Jerry Maples • U.S. Census Bureau • XXV International Methodology Symposium: Longitudinal Surveys: from Design to Analysis • Symposium international sur les questions de méthodologie • Statistics Canada • October 27-30 2009 • Gatineau, Qc, Canada

  2. SIPP Basics • National panel survey – Since 1984 with sample size between about 11,000 to 45,000 interviewed households • The duration of each panel varies from 2½ yrs to 4 yrs • The SIPP sample is a multistage-stratified sample of • the U.S. civilian noninstitutionalized population • The survey uses a 4-month recall period – 3 interviews / year • The sample is divided into 4 rotation groups for monthly interviewing • Interviews are conducted by personal visit and by decentralized telephone

  3. The Save the SIPP Campaign • New York Times Editorials • Discovering What Happens Next • Save our SIPP • The Continuing Saga of SIPP • NY Representative, Carolyn Maloney stated: • “I want to thank all of the policy groups and economists who worked so hard to help reverse the Administration’s original SIPP decision. Because of their dedication and hard work, the Administration came to understand how important SIPP is to creating and implementing good public policy.”

  4. SIPP Re-Engineering • Implement Improvements to SIPP • - reduce costs • - reduce R burden • - improve processing system • - modernize instrument • - expand/enhance use of admin records • Key Design Change: • - annual interview, 12-month reference pd., event history calendar methods

  5. EHC Interviewing • Human Memory • - structured/organized • - links and associations • EHC Exploits Memory Structure • - links between the occurrence and timing of events • EHC Encourages Active Assistance to Rs • - flexible approach to help elicit an • autobiographical “story”

  6. Evaluations of EHC Methods • Many EHC vs. “Q-List” Comparisons • - various methods • - in general: positive data quality results • BUT, Important Research Gaps • - data quality for need-based programs? • - comparison to 4-month reference period?

  7. Field Test Goals & Design • Basic Goal: • Can an EHC interview collect data of comparable • quality to standard SIPP? • - month-level data • - one 12-month ref pd interview vs. three • 4-month ref pd interviews • - especially for need-based programs • Basic Design: • EHC re-interview of SIPP sample HHs

  8. Stakeholder Concern About Test Design • Is the agreement shown in our primary re-interview results artificially inflated because of the initial interview experience – ‘priming’? • In response - two groups • Cases having answered SIPP data in the 2004 Panel about CY 2007 (Primed) • 2004 Panel cases cut as part of a sample reduction after Wave 8, only EHC data for CY 2007 (Unprimed) • Field administration and training concerns • Field observation, Training, Interviewer and Respondent Debriefings. • 3. Benchmarking to SIPP • An additional group will be created by the universe which matches to administrative records. As records are obtained for the federal (SSI, Medicare, Social Security Retirement) and state programs (TANF, Medicaid, Food Stamps, and WIC) we will continue with validation of the EHC and SIPP data for these programs.

  9. Design Details (1) • Main Sample: • SIPP 2004 Panel Cases from Waves 10-11-12 • - reported on CY-2007 via SIPP “Primed” • Supplemental Sample: • SIPP Wave 8 Sample Cut Cases • - dropped from SIPP in 2006; “Unprimed” • EHC Re-Interview in 2008, about CY-2007

  10. Design Details (2) • EHC Questionnaire • - paper-and-pencil • - 12-month, CY-2007 reference period • - selected SIPP topics (“domains”) • - start with landmark events • - within domains, anchor on “now” • - month-level (at least) detail • Field Period: Mid-April thru Late June 2008 • Sample of Addresses, Not People • - post-interview clerical match to SIPP

  11. Control Card – 2008 Field Test Instrument

  12. Page 1 of 5 – 2008 Field Test Instrument

  13. Design Details (3) • $40 Incentive, Non-Contingent and Non-FR-Discretionary • Same Response Rules as SIPP • - EHC interview for all adults (15+) • - self-response preferred (proxy permitted) • Field Staff: Census Bureau FRs • - most with some interview experience • - ~1/3 with SIPP experience • - 3-day training on EHC methods

  14. Evaluation Plans • Compare SIPP and EHC Survey Reports • - Same People • - Same Time Period • - Same Characteristics • Evaluation of “Priming” Effect • Additional Evaluations: • - Administrative Records Evaluation • - Training Evaluation • - Field Process Evaluation

  15. Design Details (4) TOTAL Initial Sample Addresses Illinois Texas Total 2007 SIPP Respondents: 487 609 1,096 SIPP Sample-Cut Cases: 427 422 849 1,945 Outcomes: Household interviews Total: 1,627 (91%) 2007 SIPP Respondents: 935 (91%) SIPP Sample-Cut Cases: 692 (91%) Individual Adult (15+) EHC interviews Total: 3,318 (99%) 2007 SIPP Respondents: 1,922 (99%) SIPP Sample-Cut Cases: 1,396 (99%) EHC Adults matched to SIPP Total: 2,756 (83%) 2007 SIPP Respondents: 1,658 (86%) SIPP Sample-Cut Cases: 1,098 (78%) FINAL ANALYSIS SAMPLE: 2007 SIPP Respondents: Total 1,620 Non-movers 1,420 Movers 200 SIPP Sample-Cut Cases: Total 1,090

  16. What were the overall research questions? • Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? • Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?

  17. What were the results? Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94% Disagreements are RARE EVENTS Results available: http://www.census.gov/sipp/DEWS/moore-aapor2009slides-abbrev.ppt

  18. Results Summary for Reinterview Evaluation 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); employment; school enrollment

  19. The Movers The ‘Main Sample (Wave 11)’ of EHC respondents who reported for CY-2007 has an important difference from our ‘Supplemental Sample (Sample Cut).’

  20. The Movers The two subsamples of EHC respondents show different characteristics, and one primary cause are the Wave 11 respondents who moved between June 2006 and mid-2007.

  21. The Movers

  22. Assessment of Priming (1) • SIPP Wave 10-11-12 Respondents • Provide CY-2007 Data Twice • - first SIPP, then EHC • - results presented previously • Are Their EHC Reports Biased by having responded to SIPP for CY-2007 already? • - e.g., more accurate EHC response • - could bias field test interpretation • Control Group: Wave-8 Sample Cut • - last SIPP response in Jun-Sep 2006 • - “Unprimed” re: CY-2007 • (however still SIPP experienced)

  23. “Priming” - SSI(Supplemental Security Income)

  24. “Priming” - WIC(Women, Infants, and Children)

  25. “Priming” - Work

  26. “Priming” - Food Stamps

  27. “Priming” - School Enrollment

  28. Priming - Findings No evidence of priming with respect to having reported about CY-2007 in SIPP. - For each characteristic - Non-Movers from W11 and Sample-Cut cases are not different - Interaction b/w these groups and months are not different - Results hold for both weighted and un-weighted models

  29. Results Implications Levels of agreement from the reinterview comparison between SIPP and the EHC - are not artificially inflated due to experience as reinterview respondents re: CY-2007 - may still be biased because both the main reinterview sample and sample-cut respondents were long-time SIPP respondents.

  30. Field Test Overall Summary Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform Next Test - larger, broader sample - “correct” timing of field period - automated questionnaire Specific Data Comparisons are Instructive

  31. Comments: • Jason Fields – Jason.M.Fields@Census.Gov • Jeff Moore – Jeffrey.C.Moore@Census.Gov

  32. Assessing Users’ Needs URL: http://www.sipp.census.gov/sipp/dews

  33. Page 2 – 2008 Field Test Instrument

  34. Page 3 – 2008 Field Test Instrument

  35. Page 4 – 2008 Field Test Instrument

  36. Page 5 – 2008 Field Test Instrument

More Related