ScWk 240 Week 6

1. ScWk 240 Week 6 Measurement Error Introduction to Survey Development “England and America are two countries divided by a common language.” George Bernard Shaw ScWk 240 Week 6

2. Review--Variables: • When we use the word “outcome” are we talking about the independent variable or the dependent variable?___________ • In an evaluative study, the primary independent variable is the __________ • “This study examines the impact of acculturation on the use of health services by immigrants, controlling for income, age, and health status. • What is the dependent variable? ___________ • What is (are) the independent variable(s)? _______________________________________________________________ ScWk 240 Week 6

3. Review: Conceptualization • We’re talking about complex ideas • Depression • Self esteem • Acculturation • How would you conceptualize “acculturation”? • Give a one sentence conceptual definition • What dimensions would you include? ScWk 240 Week 6

4. Operationalization – the next step • “Operationalization” – the process of defining a specific variable, and identifying the relevant attributes of the variable • Example – Depression “loss of energy” dimension • Variable: “loss of energy” • Measured by: (CIDI instrument) During the period when your sadness was most severe and frequent, did you feel tired or low in energy even when you had not been working very hard? • Attributes: • Yes • No • Don’t know ScWk 240 Week 6

5. Now operationalize “acculturation” • Pick one dimension • For that dimension, describe the variable in a simple statement • First, give the variable a label (name) in 1-3 words • Include in the simple statement “…as measured by…” • For that variable, identify one set of attributes ScWk 240 Week 6

6. Small group exercise: • Pick a concept as a group • Identify three dimensions of that concept • Choose one dimension • For that dimension, identify and describe one variable in a simple statement • First, give the variable a label (name) in 1-3 words • Include in the simple statement “…as measured by…” • For that variable, identify one set of attributes ScWk 240 Week 6

7. Student Report What is “culturally competent measurement” in research? Relate this to one or more (of the 5) areas of the Transcultural Perspective. Elisse ScWk 240 Week 6

8. “Error” in Measurement • What is “measurement error”? Measurement error occurs when we obtain data that do not accurately reflect the concept we want to measure. • There are two main sources of measurement error: • Random error: inconsistent error. There’s no pattern or predictability • Systematic error: consistent error. There is pattern and predictability ScWk 240 Week 6

9. Random Error – inconsistent response • Environmental factors: i.e. surveys administered in a hot room will vary from surveys administered in a more comfortable setting • Questions with jargon or technical terms will be interpreted differently by different respondents (or other factors related to a poorly constructed instrument) • Written survey response will differ by educational level or reading ability of respondents ScWk 240 Week 6

10. Systematic error—consistently and predictably “wrong” response • Social desirability bias’ – respondents portray themselves in the best light • ‘Acquiescent response set’ – Agree or disagree with everything • Cultural ‘bias’ – cultural definitions of concepts different than those intended by researcher (i.e. you can always expect physical somatization as a stated primary symptom of depression in many non-Western cultures) • Bias from wording of questions – i.e. “Are you resistant to treatment?” ScWk 240 Week 6

11. Review: Potential for systematic or random error? • People are more alert in the morning than after lunch ____________ • The meaning of “living independently” varies cross-culturally __________ • Male respondents tend to minimize their aggressive tendencies _________ • A 16 y.o. taking a test: “I just circle ‘C’ each time and hope for the best” (hint-this is not Acquiescent Response) _________ • The person entering data is not careful enough and mis-codes some of the responses ________ ScWk 240 Week 6

12. Student reports • What’s the difference between “reliability” and “validity” of a measure? Glorialee • How is validity related to cultural issues? Dawei ScWk 240 Week 6

13. Measuring error—Validity and reliability • The presence of error is not “all or nothing” – there’s always going to be some error • We have to measure it to understand its impact • Reliability – measures random error—consistency of meaning • Validity – measures systematic error—accuracy of meaning ScWk 240 Week 6

14. Reliability • Concerned with whether or not a particular technique done repeatedly would yield the same result each time • Inter-rater reliability—Two researchers rating the same behavior: do they rate them the same? • Test-retest reliability—Is the response from the first test the same as the response from a re-test? • Alternate forms reliability—Is there the same response from slightly different versions of same measure? • Internal consistency reliability & use of coefficient alpha—Are similar questions (about the same concept) in survey are answered in the same way? ScWk 240 Week 6

15. You‘ll see this in your lit review… • Methods section—measures used: the author will report on reliability testing (if done) using the coefficient alpha • A number that tells you, overall, how well the questions in an instrument measuring the same concepts correlate with each other (internal consistency reliability) • A coefficient alpha is considered good if it is .80 or above • The author will then report “The instrument showed good internal consistency reliability” ScWk 240 Week 6

16. Validity • Concerned with the accuracy of meaning—are you measuring what you originally intended? • Types of validity • Face validity – experts’ understanding of concept • Content validity – Does measure cover all relevant dimensions? • Criterion-related validity – Does measure of concept agree with (or relate with) another related concept (external criterion)? • Construct validity – Does measure agree with or correlate with measures of other theoretically related concepts? ScWk 240 Week 6

17. Validity examples • Validity studies show that a set of questions about alcohol dependence correlates well with the observed daily amount of alcohol consumption. Criterion-related validity • Most experts would agree that children with Attention Deficit Disorder with Hyperactivity have difficulty staying on tasks (even those the child likes). Face validity • Couples who test as having low levels of marital satisfaction communicate less effectively, leading to high levels of conflict and, eventually, divorce. Construct validity • The Beck Depression Scale covers those dimensions of depression that are most supported by evidence. Content validity ScWk 240 Week 6

18. Student reports for next week • 1. Dawei • What are some strategies for ensuring reliability when developing survey questions?___Amanda • What are some strategies for ensuring validity when developing survey questions?____Sarah • What is the main difference between “open-ende” and “closed-ended” questions? Selene ScWk 240 Week 6

19. Concepts to know • Measurement error • Random error • Systematic error • Reliability (Inter-rater, Test-retest, Alternate forms, Internal consistency) • Coefficient alpha • Validity (Face, Content, Criterion-related, Construct) ScWk 240 Week 6