1 / 126

A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar:

Explore the efficiency and accuracy of survey data obtained using standard questionnaire versus event history calendar methods. Initial findings from the 2008 EHC "Paper" Test presented at ASA/SRM SIPP Working Group. Comparison of data quality, memory structure exploitation, and active respondent assistance. Evaluation of EHC methods and important research gaps discussed.

sames
Download Presentation

A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar: • Initial Results from the 2008 EHC “Paper” Test • Jeff Moore • Statistical Research Division • Presentation to the ASA/SRM SIPP Working Group • Alexandria, VA • November 17, 2009

  2. Paper prepared by: • Jeff Moore, Jason Fields, Gary Benedetto, • Martha Stinson, Anna Chan, & Jerry Maples • For presentation at: • American Association for Public Opinion Research (AAPOR) • May 14-17, 2009

  3. Overview Background: - SIPP “re-engineering” - event history calendar (EHC) methods Goals & Design of the 2008 EHC Paper Test Preliminary Results Summary / Conclusions / Next Steps

  4. SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years

  5. SIPP Survey of Income and Program Participation - income/wealth/poverty in the U.S.; program participation dynamics/effects - interviewer-administered; longitudinal - panel length = 3-4 years Key Design Feature: - 3 interviews/year, 4-month reference pd.

  6. SIPP Re-Engineering • Implement Improvements to SIPP

  7. SIPP Re-Engineering • Implement Improvements to SIPP • - reduce costs • - reduce R burden • - improve processing system • - modernize instrument • - expand/enhance use of admin records

  8. SIPP Re-Engineering • Implement Improvements to SIPP • - reduce costs • - reduce R burden • - improve processing system • - modernize instrument • - expand/enhance use of admin records • Key Design Change: • - annual interview, 12-month reference pd., event history calendar methods

  9. SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods

  10. EHC Interviewing • Human Memory • - structured/organized • - links and associations

  11. EHC Interviewing • Human Memory • - structured/organized • - links and associations • EHC Exploits Memory Structure • - links between to-be-recalled events

  12. EHC Interviewing • Human Memory • - structured/organized • - links and associations • EHC Exploits Memory Structure • - links between to-be-recalled events • EHC Encourages Active Assistance to Rs • - flexible approach to help elicit an • autobiographical “story”

  13. Evaluations of EHC Methods • Many EHC vs. “Q-List” Comparisons • - various methods • - in general: positive data quality results

  14. Evaluations of EHC Methods • Many EHC vs. “Q-List” Comparisons • - various methods • - in general: positive data quality results • BUT, Important Research Gaps • - data quality for need-based programs? • - extended reference period?

  15. Paper Test Goals & Design • Basic Goal: • Can an annual EHC interview collect data of • comparable quality to standard SIPP?

  16. Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D

  17. Paper Test Goals & Design Basic Goal: Can an annual EHC interview collect data of comparable quality to standard SIPP? “Go/No-Go” signal for continued R&D Basic Design: EHC re-interview of SIPP sample households

  18. Design Details (1) • Sample: • SIPP 2004 panel interview cases • - reported on CY-2007 in waves 10-12

  19. Design Details (1) Sample: SIPP 2004 panel interview cases - reported on CY-2007 in waves 10-12 EHC re-interview in 2008, about CY-2007

  20. Design Details (2) • SIPP Sample Cases in Two Sites • - Illinois (all) • - Texas (4 metro areas)

  21. Design Details (2) • SIPP Sample Cases in Two Sites • - Illinois (all) • - Texas (4 metro areas) • Primary Sample Component: • 1,096 Wave 10-11-12 Addresses • (cooperating wave 11 households) • IL: 487 • TX: 609

  22. Design Details (3) • EHC Questionnaire [handout]

  23. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period

  24. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events

  25. Design Details (3) EHC Questionnaire [handout] - paper-and-pencil - 12-month, CY-2007 reference period - start with landmark events - subset of SIPP topics (“domains”) - month-level detail

  26. Design Details (3) • EHC Questionnaire [handout] • - paper-and-pencil • - 12-month, CY-2007 reference period • - start with landmark events • - subset of SIPP topics (“domains”) • - month-level detail • Sample of Addresses, Not People • - post-interview clerical match to SIPP

  27. Design Details (4) • 1096 initial sample addresses

  28. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%)

  29. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%)

  30. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%)

  31. Design Details (4) 1096 initial sample addresses Outcomes: - 935 household interviews (91%) - 1,922 individual EHC interviews (99%) - 1,658 EHC Rs matched to SIPP (86%) FINAL ANALYSIS SAMPLE: 1,620

  32. Primary Evaluation • Compare SIPP and EHC Survey Reports

  33. Primary Evaluation • Compare SIPP and EHC Survey Reports • - same people • - same time period • - same characteristics

  34. Primary Evaluation • Compare SIPP and EHC Survey Reports • - same people • - same time period • - same characteristics • Differences Suggest Data Quality Effects

  35. Primary Evaluation • Compare SIPP and EHC Survey Reports • - same people • - same time period • - same characteristics • Differences Suggest Data Quality Effects • (later: use administrative records for a more • definitive data quality assessment)

  36. Main Research Questions • Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)?

  37. Main Research Questions • Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? • Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)?

  38. Main Research Questions • Are responses to Qs about government programs and other characteristics affected by interview method (SIPP vs. EHC)? • Does the effect of interview method vary across calendar months (especially early in the year vs. late in the year)? • 3. Bottom line: Go? Or no-go?

  39. Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC)

  40. Initial Results 4 Government “Welfare” Programs: Food Stamps Supplemental Security Income (SSI) Temporary Aid to Needy Families (TANF) Women Infants & Children (WIC) 4 Other Characteristics: Medicare Social Security employment school enrollment

More Related