1 / 22

William van der Veld (University of Amsterdam) Willem Saris (ESADE Business School)

THE NATURE OF MEASUREMENT ERROR IN PANEL DATA Reliability and Opinion Stability. William van der Veld (University of Amsterdam) Willem Saris (ESADE Business School) Barcelona, 2005, European Association for Survey Research. Overview. The RUSSET panel (www.vanderveld.nl)

coby
Download Presentation

William van der Veld (University of Amsterdam) Willem Saris (ESADE Business School)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. THE NATURE OF MEASUREMENT ERRORIN PANEL DATA Reliability and Opinion Stability William van der Veld (University of Amsterdam) Willem Saris (ESADE Business School) Barcelona, 2005, European Association for Survey Research

  2. Overview • The RUSSET panel (www.vanderveld.nl) • Reliability estimation in a panel design • Reliability estimation in a cross-section design • Comparison of Panel & Cross-section • The VAS Model • Conclusion

  3. The RUSSET panel Aggregate level [means] Individual level (Correlations)

  4. Reliability estimation in a panel design Interpretation • y: The observed variable • e: the random error in y • T = y + e • When T and y standardized, • β: the stability coefficient • λ: reliability coefficient Quasi-simplex model: Assumptions • The random error variance is the identical for the repeated measures; • For each respondent the attitude changes according to a lag-1 quasi-simplex; • The repeated measures are independent.

  5. Reliability estimation in a panel design • Estimation of Reliability & Stability • RUSSET Data: 3 waves, 3 times the question: • How satisfied are you with your current life…? • n=837; Chi-square=0; df=0 [Lisrel, ML estimation]

  6. Reliability estimation in a panel design • The observed stability is:0.34 & 0.27. • After correction for measurement error, the stability is:0.99 & 0.82. • The reliability coefficient of a simple question as ‘How satisfied are you with your current life as a whole?’ is quite poor: ±0.6 (reliability=0.36) • Is there an alternative way to estimate the reliability,so that we can compare both estimates?

  7. Reliability estimation in a cross-section design Parallel-test model: Assumptions • The random error variance is the identical for the repeated measures; • For each respondent the attitude has not changed during the interview; • The repeated measures are independent. Interpretation • y: The observed variable • e: the random error in y • T = y + e • When T and y standardized, • λ: reliability coefficient

  8. Reliability estimation in a cross-section design • Estimation of Reliability (not Stability) • RUSSET Data: 1 wave, 2 times the question: • How satisfied are you with your current life…? • Same respondents (exactly), same question. • n=837; Chi-square=1; df=1 [Lisrel, ML estimation]

  9. Reliability estimation in a cross-section design • THAT’S STRANGE!!!! • Same question, same respondents, same moment • Same definition: • Still estimated reliability PTR(6)=0.85 vs. QSR(6)=0.58 • No confirmation because they are different. • So, this result does not help • Is there a way to check which reliability is correct?

  10. Comparison of QSR & PTR • Use the reliability-estimates to correct the observed correlation between different variables for measurement error. • ρ21 = r21/(λTx1* λ Tx2) • We have observed the correlation: r21 • We have estimated the reliability coefficients with QS & PT model. • We are interested in: ρ21 • Which ρ21 is more plausible?That obtained with the QS-estimates or that with the PT-estimates

  11. Comparison of QSR & PTR • Exactly the same data (n=837), Wave 1995(6,7). • Correction of the observed correlation between:Satlife & Satinc. • Satlife:How satisfied are you with your current life as a whole? • Satinc:How satisfied are you with your family’s current financial situation? • Observed correlation between Satlife & Satinc = 0.51 • The reliability coefficient estimates are:

  12. Comparison of QSR & PTR

  13. Comparison of QSR & PTR

  14. Comparison of QSR & PTR

  15. Comparison of QSR & PTR • It appears that the QSR-estimates are wrong. What’s wrong with the quasi-simplex reliability? • The key is provided by the VAS model (Van der Veld & Saris, 2004) • It can be derived that: QSR=q(vas)*c(vas) • This is proven in the accompanying paper, but it can be illustrated too.

  16. The VAS Model For details, visit Session:Nonattitudes and informed opinions at Thursday 11:30Chair: Peter Neijens

  17. The quasi-simplex model

  18. The VAS Model For details, visit Session:Nonattitudes and informed opinions at Thursday 11:30Chair: Peter Neijens

  19. The VAS Model - Estimation of Q, S, & C • RUSSET Data: 3 waves, 6 times the question (n=627). • Satlife:How satisfied are you with your current life as a whole? • Satinc:How satisfied are you with your family’s current financial situation?

  20. Comparison of QSR & PTR & VAS • QSRC=q*cQSRCsatlife=.81*.75=.60QSRCsatinc=.78*.70=.54

  21. Conclusion • The quasi-simplex model has parameters for • Stability • Reliability • The reliability in that model is not correct, It is the product of: • The quality of the measurement instrument • The opinion crystallization • The stability is correct [see paper].

  22. Conclusion • How do you feel about public security these days? • With the QSM all the considerations that are unique for a specific time are not in the variable S. Hence such considerations are not part of the stability. • One should be aware of this. • It depends on the object of study, whether this model is appropriate to estimate the stability! • It is definitely not correct for reliability estimation!

More Related