The problem of identifying persons with disabilities – the importance of questionnaire design

1. The problem of identifying persons with disabilities – the importance of questionnaire design Angela Me, Chief Social and Demographic Statistics Section

2. Challenge • How to measure a wider experience of disability through a limited number of questions?

3. Design an instrument to identify the defined population with disability • The difficult part is: To logically convert/translate objectives into measurement instruments and to link definitions with questions

4. Design an instrument to identify the defined population with disabilities - a Census Example • Definition: • Any restriction or lack (resulting from an impairment) of ability to perform an activity in the manner or within the range considered normal for a human being • Question: • Is there anybody in the household who is disabled?

5. Definition-measurement instrument BFS limitations Paralyzed ….. Deaf Activity limitations …..

6. Design an instrument to identify the defined population with disabilities - a Census Example Despite a definition based on activity limitations, the questions identified only persons with most severe impairments 2.5%

7. From theory to practice • Estimates of prevalence of disability are highly sensitive to the measures used

8. Measurement Issues • Design of the study • Method of data collection • Question design (wording, place, length, …) • Interview process: • interviewer effect • Respondent effect • Socio-cultural Determinants

9. US Survey Example The following questions and results were obtained in an American survey % 'Yes' Have you ever heard the word AFROHELIA? (no such word!) 8 Have you ever heard of the famous writer, John Woodson? (no such writer!) Have you ever heard of the Midwestern Life Magazine? (no such magazine!) Do you recall that, as a good citizen you voted last December in the special election for your state representative? (no election!) 16 25 33 Have you ever heard of the Taft-Pepper Bill concerning veteran's housing (no such bill!) 53

10. Example: Australian Survey Average number of sex partners reported • By women who were watched as they filled in their survey answers: 2.6; • By women who knew they were completely anonymous: 3.4; • By women who thought they were attached to a lie detector: 4.4 Sydney Morning Herald, August 31, 2003

11. Wording • The most detailed disability survey, using a carefully designed and relatively complete set of questions covering a wide range of topics, is limited when the initial questions used to identify the persons with disability is poorly designed

12. Developing instruments to identify persons with disabilities Disability is a dynamic complex related to: • Individual attributes • Environment • Time Two persons with the same impairment may have a different perception of disability

13. Issues that we need to consider Particular attention is needed to measure disability through an interview process • People may be unwilling to talk about their problems • Difficulty in defining what is meant by disability and its various aspects • Stigma

14. Issues that we need to consider In an interview process: • Easier to measure activity limitations (day-to-day activities) and participation

15. Developing instruments to identify the complexity of disability • Requirement: • Multiple questions to set context, clarify terminology, define multiple domains • Resource availability • Short questions

16. Developing instruments to identify the complexity of disability • Long instruments/modules • High number of questions, more opportunities to capture the different dimensions, intensity • Short instrument/modules • 1-5 questions to identify persons with disabilities • Careful design of the question(s) to make sure that all persons with disabilities that we want to identify can indeed be properly identified

17. What defines a good instrument • The instrument measures the concept it is supposed to measure (Validity-Accuracy) • Repeated measurements of the same instruments give the same results (Reliability-Precision)

18. Design an instrument to identify the target population- A Census Example • Definition: • Any restriction or lack (resulting from an impairment) of ability to perform an activity in the manner or within the range considered normal for a human being • Question: • Is there anybody in the household who is disabled?

19. An example: U.K. Census 1991 • Do you have any long-standing illness, health problem or handicap which limits your daily activities or the work you can do? Include problems which are due to old age

20. Conditions that effect the output of a question • Wording • Context • Self/Proxy • Response categories • Mode of data collection • Method of data collection • Overall survey topics • Survey sponsor

21. Wording: what does affect comprehension? • Ambiguous syntax • Complicated syntax • Unfamiliar terms • Vague concepts • Assumptions about respondent’s knowledge

22. Wording Language: • Clear • Unambiguous • Simple Terms such as long-term, disabilities, handicaps are viewed as extremely negative and tend to underreport disabilities (Langlois, 2001)

23. Because of a physical, mental, or emotional conditionlasting 6 months or more, does this person have any difficulty in doing any of the following activities… d. Workingat a job or business? Multiple health domains (explicit enumeration) Duration “Difficulty” capacity participation “Working” 2000 US Census

24. Question components for a short question • Preamble • Health condition (as cause) • Duration (long/short term disability?) • ICF domain • Functioning • Activities • Participation

25. Question components for a short question Keep it relevant and valid but SIMPLE • If possible split the different components of the measure into different questions

26. Question components for a short question Introduction: make the respondent think about the outputs of an health condition and set the duration (conditions that last for 3 months, 6 months, 12 months, …) Depending on the domain we want to identify: • Do you have difficulties walking? • Do you have difficulties concentrating? • Do you have difficulties participating in daily activities • …..

27. Question components for a short question Questions based on activities and participation rather than impairments provide a broader view of disability and they facilitate the identification of persons with disabilities by providing a more simple and natural language

28. Response process: judgment and response formulation • Evaluation of retrieved information • Perception of accuracy • Motivation • Evaluation of response options • Communication of response

29. Response categories • Response categories set the context of the question • Avoid to force the respondent to identify him/herself with a socially-defined label (stigma) • Scale Response instead of a dichotomy • None/A little/A lot • Yes, sometimes/Yes, often/No

30. Response categories • It has been proven that scaled responses improve the respondents’ ability to report having disabilities (Statistics Canada, Austrian Bureau of Statistics, Research in the USA) • If the respondent can not choose among multiple dimensions, he/she is likely to misreport his/her disability status Disability is not a yes/no phenomenon but rather a status that varies on a continuum in terms of intensity and time

31. Mode of data collection • Self reported or Proxy? • Avoid if possible proxy responses • The disability process relates to the individual’s experience and can be accurately described only by the individual him/her self

32. Type of question • Avoid household-based questions (Is there anybody in the household who has difficulties walking?) • Use person-based question (Do you have difficulties walking?) • It has been proven that person-based questions identify more persons with functional limitations (USA, UN)

33. Context • Place of the question in the questionnaire • Example: disability and economic characteristics questions • Introduction to the question