Validation in Statistics Canada Health Surveys

1. Validation in Statistics Canada Health Surveys Presentation to RRFSS Workshop June 20, 2007 Vincent Dale

2. Outline • Statistics Canada quality assurance framework • Ensuring data accuracy • Past validation projects • Future projects • Future directions

3. Quality Assurance Framework

4. Statistics CanadaQuality Assurance Framework • Trade-offs between aspects of quality • These are actively managed through a variety of processes, including: • User and stakeholder feedback mechanisms • Program review • Data analysis and dissemination • Standards and documentation (concepts, variables, classifications)

5. Ensuring Data Accuracy • Questionnaire development • Wherever possible, validated questionnaire modules are used • Sometimes modified for use in population-based survey • Sometimes not as valid as advertised • Questionnaire testing • STC policy requires testing of all new questionnaires • Cognitive interviews and focus groups • Coherence versus accuracy • Sometimes better to keep measure stable even if imperfect

6. Ensuring Data Accuracy • Sampling error • error attributed to studying a fraction of a population rather than carrying out a census • Non-sampling error • coverage errors • response errors • non-response errors • processing errors • estimation errors • analysis errors

7. Ensuring Data Accuracy • Explosion of health survey data • More data, more often for smaller levels of geography • Increasing attention paid to validity • Health measures • Administrative data • Complimentary surveys

8. What is validity? • Face validity • Internal validity • construct validity • External validity • Criterion • Sensitivity, specificity, predictive value

9. Past CCHS Validation Projects

10. Health Care Utilisation • Data linkage of CCHS responses with BC administrative health records • Supplemented with analysis of: • Respondent interpretation and formulation of responses • Interviewer behaviour and training • Patterns in response changes, edits and timing of response entry

11. Contacts with Health Professionals • Results of linkage: • Compared to provincial health records: • Most CCHS respondents (58%) reported fewer primary care physician contacts • On average, CCHS respondents reported 1.7 fewer primary care physician contacts • Older CCHS respondents and respondents with better self-perceived health tended to report fewer contacts • Younger respondents and respondents with poorer self-perceived health tended to report more contacts

12. Contacts with Health Professionals • Recommendations from study: • Revise wording of specific questions to minimize misinterpretation • Facilitate consistent interviewer probing techniques • Improved edits and CAPI/CATI application navigation for interviewers to facilitate changes to previously-answered questions

13. Evaluation of coverage of linked CCHS and hospital inpatient records • Probabilistic linkage used to identify CCHS 1.1 respondents (excluding Québec) hospitalized over a 14-month period • Health person-oriented information database (HPOI) is a virtual census of hospital admissions and used as the standard • Survey weights applied to the 8230 CCHS records which were found in the HPOI database

14. Evaluation of coverage of linked CCHS and hospital inpatient records

15. Evaluation of coverage of linked CCHS and hospital inpatient records • Under-reporting rates similar between women and men • Lower among Manitoba residents (69.2%) • Higher among individuals aged 12-74 (86.1%) than those aged 75+ (70.3%) • Under-reporting is an essential prerequisite to further analyses based on the CCHS – HPOI linked data • Use of the linked file could lead to bias due depending on province/territory of residence and age

16. CCHS Measured Height & Weight • In 2005, height / weight were measured for a sub sample of CCHS Cycle 3.1 participants (n=4567) • Weight: mean difference between measured and self-reported weight of 2.1 kg (2.5 kg for women) • Height: mean difference between measured and self-reported height of -0.7 cm (-1.0 cm for men) • BMI: mean difference between measured and self-reported BMI was 1.1

17. CCHS Measured Height & Weight

18. CCHS Mode Effect Study • Potential differences associated with two methods of collection used in CCHS • CAPI: computer assisted personal interview • CATI: computer assisted telephone interview • Used a split-panel design with a unique sample frame • secondary sampling units randomly assigned to CAPI or CATI. • Fully integrated as part of CCHS cycle 2.1 • 11 sites selected to provide a good representation of each region in Canada

19. CCHS Mode Effect Study • Important differences observed for obesity rates • CAPI = 17.9%; CATI = 13.2% • Physical activity index – inactive persons • CAPI = 42.3%; CATI = 34.4% • Statistically significant differences for contact with medical doctors and unmet health care needs • No significant differences observed in the vast majority of health indicators

20. CCHS Mode Effect Study • Overall results show that cycles 1.1 and 2.1 are largely comparable despite an increase in CATI collection for Cycle 2.1 (2003) • Results led to a decision to measure exact height and weight for a sub-sample of respondents in cycle 3.1 (2005) • Led to improved standardization of interviewer procedures across the two collection modes

21. Future Validation Projects

22. Scale Reliability - Factor Analysis • Construct validity / scale reliability: • Cronbach’s Alpha calculated for scales used in CCHS questionnaire • Results could be published in user guide • What are standards? • Some researchers feel that scores should be above 0.8

23. CCHS Depression Module • Currently, CIDI Short form for Major Depression (CIDI-SF) is used in CCHS • Also used in NPHS and several regional and provincial surveys • Some problems with its use in CCHS • Has not been validated against International Classification of Disease (ICD) • Evaluates 12-month prevalence, not necessarily current treatment need • Does not evaluate some items related to clinical significance • Patient Health Questionnaire (PHQ) identified as potential CIDI-SF replacement

24. CCHS Depression Module • Primary goals of potential validation study: • Determine the validity of the CIDI-SF and PHQ in relation to a gold standard diagnostic interview (SCAN – Schedules for Clinical Assessment in Neuropsychiatry) • Identify optimal scoring procedures for the PHQ in Canadian population-based studies

25. CCHS Depression Module • Samples of n=200 subjects to be drawn in two sites (English and French) • Supplemented with n=100 subjects selected from psychiatric outpatient settings to increase the number of positive cases of major depression • Each participant to be administered: • 1) Standard demographic module • 2) PHQ-9 • 3) Module to distinguish between clinical depression and bereavement • 4) SIDI-SF • 5) Set of modules to assess consequences of construct in terms of quality of life

26. CCHS Depression Module • Sensitivity and specificity of the CIDI-SF and PHQ to be measured using the SCAN as a gold standard • Ordinal CIDI-SF ratings to be correlated with PHQ ordinal ratings using Spearman correlation coefficient • Test of construct validity of PHQ to be performed using exploratory factor analysis • Internal consistency of scales and subscales to be assessed using Cronbach’s alpha • Test-retest reliability of PHQ and CIDI-SF and inter-rater reliability of the SCAN will be evaluated for 50 respondents

27. CCHS Depression Module • The estimated cost for the project exceeded $200,000 • Due to our inability to secure external funding and the lack of available budget and personnel internally, there are no concrete plans to proceed with study

28. Directions Forward • Focus on accuracy, interpretability and coherence • Trade-offs between aspects of data quality • Improved timeliness, accessibility and relevance • How good is “good enough”? • Partnerships • Are there areas where CCHS, RRFSS and others can collaborate ?

29. Contact Information Vincent Dale Survey Manager, Canadian Community Health Survey 613-951-4265 Sylvain Tremblay Content Manager, Canadian Community Health Survey 613-951-2528