Social Welfare 580 Advanced Research Methods & Design

1. Social Welfare 580Advanced Research Methods & Design Measurement Instrument Design Psychometrics

2. Dr. Caprice Hollins, the director of equity and race relations for Seattle Public Schools. “We know that cultural competence is an important aspect of being an effective teacher, but how do you measure that?” “Thus, no research has shown that teacher’s cultural competence is associated with better student learning.”

3. Five topics • measurement theory • reliability • validity • measurement selection • measurement construction

4. measurement theory • levels of measurement • N O I R • Consequences of level of measurement on analysis choices

5. measurement theory • “fundamentalist” or representational view:Stevens: • measurement scales are direct representations of what they measure • a measure of an attribute can be classified into distinct levels (N,O,I or R) • analyses are only permitted as limited by the level of measurement

6. measurement theory a fundamentalist “map” of permissible analyses

7. measurement theory • operational or statistical viewNunnally, Gaito, etc.: • a score does not represent the trait, it is just a measure of the trait • but each score has error and is thus only an estimate • permitted analyses depend on the set of items used to minimize error (reliability) as well as considerations for the underlying nature of the phenomenon being measured (intrinsically N, O, I R?

8. measurement theory e e e e e unique measurement variance item 5 item 4 item 3 item 2 item 1 construct common measurement variance

9. measurement theory • Under operational theory… • Likert and other similar scales fundamentalists would consider ordinal can be analyzed using parametric procedures if • the set of items reflect a reliable assessment of the construct and the underlying concept under study is likely interval or ratio. • In doing this we assume that the numbers we assign the ranked categories approximately assess the true (but unknown) intervals between them.

10. reliability Does the measure give consistent results? • inter-rater • test-retest • parallel-forms • internal consistency

11. validity Is the measure assessing what it claims? • face • content • criterion (predictive & concurrent) • construct

12. Measurement selection and development

13. Four typical modes of measurement in social research • archival assessment – example FirstSteps Database • self-report/self-administered – example from Ages and Stages Questionnaire • interviews – example from SSDP Interview • behavioral observation - example: TIP Observational Coding System

14. SSDP Intergenerational ProjectSocial Development Research Group Karl G. HillPrincipal Investigator J. David HawkinsInvestigator Richard F. CatalanoInvestigator Robert McMahonInvestigator Lewayne GilchristCollaborator Jennifer BaileyResearch Analyst Sabrina OesterleResearch Analyst Maria Roper-CaldbeckData Collection Supervisor Karen SegarData Manager Amber TabaresGraduate Research Assistant Erica BourgetProgram Coordinator

15. Method & Measures TIP Sample • The SSDP sample parent, • His or her eldest biological child, • A second primary caregiver (most often the spouse) identified by the sample parent as the person who shares the greatest responsibility for raising the child. Procedure • Year-round data collection where the child and caregivers are interviewed +/- six weeks around the child’s birthday. • In-person interview and observation session consisting of: • parent interview • parent child-interaction task (age 2+) • other caregiver interview • other caregiver-child interaction task (age 2+) • child interview (age 6+)

16. TIP descriptives • G3 Sample Size • 258 Families/children • 173 SSDP mothers, 85 SSDP fathers • 132 girls and 107 boys • 40% age 1-5 • 60% age 6-13 95% retention in Waves 2 and 3 • G3 Ethnicity • 33% Caucasian American • 21% African American • 37% Multiethnic • 9% Asian, Hispanic, Native American • Family Structure • 70% two parents (95% of whom live together), • 20% were being raised by one parent alone, • 10% one parent and a non-parental caregiver.

17. Selecting pre-existing measures Research the literature to determine how your constructs have been measured before. Try to use established measures when possible. Note: mode length/#items reliability & validity info Obtain copies for pretesting Alter the assessments only if absolutely necessary. Identify and define the constructs you want to measure

18. Selecting Pre-existing Data:Archival Assessment– example FirstSteps Database The database links measures of pregnancy outcomes to descriptions of maternity care services (from Medicaid payment records for Medicaid clients in Washington State) and background information on the mother's health and socio-demographic status (from birth and death certificates for all Washington residents).

19. Archival Assessment – example FirstSteps Database

20. Archival Assessment Other kinds of archival records Individual • school records (grades, achievement test, disciplinary action, etc.) • police, court & detention records • health? Aggregate • schools, neighborhoods, etc.

21. Selecting pre-existing measures These guidelines should be kept in mind in selecting pre-existing measures: • What level of measurement does it provide? • Possible Bias? • Is it reliable? • Has it been validated?

22. Self-report/Self-administered AssessmentAges and Stages Questionnaire (TIP)

23. Self-report/Self-administered AssessmentAges and Stages Questionnaire (TIP) • DEVELOPMENT OF THE AGES & STAGES QUESTIONNAIRES • Item Selection • ASQ items were developed using a variety of sources, including standardized • developmental tests, nonstandardized tests focused on early development, • textbooks, and other literature containing information about early developmental • milestones. Using these sources, the following criteria were used to • develop items: • 1. Skills were selected that could be easily observed or elicited by parents. • 2. Skills were selected that were highly likely to occur in a home setting. • Once skills were selected, items were written in familiar, nonjargon • wording not to exceed a sixth-grade reading level, illustrations were provided • when possible, and concrete examples were provided as appropriate.

24. Self-report/Self-administered AssessmentAges and Stages Questionnaire (TIP)

25. Self-report/Self-administered AssessmentAges and Stages Questionnaire (TIP)

26. Scale/item/measure development:when there are no good pre-existing measures of the construct. broad qualitative studies focusgroups togenerateitems final scaleimplementation theory /research /existing measures item and scale construction pretesting &reliability &validity analyses

27. Constructing Measurement Instruments (Schaeffer & Pressler 2003; R&B Ch 7) • Guidelines for asking questionsquestions/statements; clarity; avoid double-barreled items; reading level; relevant; short; phrase in positive; avoid bias; • Questionnaire construction • Composite Measures • Scaling Procedures

28. Example Structured InterviewSeattle Social Development Project Work Prosocial involvement Prosocial rewards Bonding to prosocial others Prosocial Beliefs External constraints:Norms Family & Classroom Management Positive Behavior Skills for interaction Position in the social structure: race, SES, age, gender Problem Behavior Individual constitutional factors Antisocial opportunities Antisocial involvement Antisocial rewards Bonding to antisocial others Belief in antisocial values ANTISOCIAL PATH ( +, - ( - ) ( + ) Social Development Model PROSOCIAL PATH Prosocial opportunities

29. Interviews – example from SSDP Interview Work Opportunities • There are a lot of chances for people like me to get a job with a future. • There are a lot of chances to move ahead or advance in my job. • These days it's just about impossible for someone like me to find a job. • The only jobs available for someone like me are dead-end jobs. • I have a lot of chances to do things with my coworkers. • I have lots of chances to do new things in my job. • I have lots of chances to make a contribution in my job. Work Involvement • How often are your involved in making significant decisions at work? • How often do you interact, or talk with your coworkers? • How often do you supervise others on your job? • How much responsibility do you have in your job? [Would you say…] • I put a lot of energy into my job. Work Rewards • You feel that you are making a useful contribution to society. • I gain a sense of accomplishment from my job. • My supervisors recognize when I am doing a good job and let me know about it. • My job is rewarding.

30. Scaling Procedures SPSS example • Evaluation of items • Reliability Analysis • Validity Analysis

31. Reliability and Validity Analyses: see handouts correlations Construct alpha 1 2 3 1 Work Opportunities .77 - 2 Work Involvement .66 .45 - 3 Work Rewards .80 .57 .47 -

32. Interviews • Semi-structured (interviewer has a general structure and common prompts to move the interview along to elicit themes of interest to the study). Used in more qualitative studies. • Structured(interviewer has a determined structure of questions to ask, prompts, etc.)

33. SSDP Structured Interview: Age 24Work Opportunities, Involvement, & Rewards

34. SSDP Structured Interview: Age 24Work Opportunities, Involvement, & Rewards

35. Observational CodingAdvantages of Observational Coding • Theorywitness behavior unavailable to other methods • Analysismulti-modal assessment to reduce bias • Prevention/Interventionobservations are tied more closely to the identification of specific behaviors that could be targeted for intervention

36. Social Development Model—Observational Coding System • Theory driven • Observed, in-home, parent-child interactions • For families with children age 2 to 18 yrs • Demonstrated predictive validity

37. The Social Development (SDM)Observational Coding System • Two coding systems have been developed • One to code families with children 2-8 years old. • Another to code families with children ages 9+. • Parallel SDM constructs are used in both systems. • Micro- and macro-level ratings are included in the SDM-OCS.

38. These macro-coded scales reflect SDM and other constructs of interest External Constraints • prosocial guidelines • antisocial guidelines • coercive parenting Opportunities • prosocial opportunities Involvement • dyad’s involvement in the activity • parent’s prosocial involvement • parent’s antisocial involvement • child’s prosocial involvement • child’s antisocial involvement • dyad’s positive involvement Consequences • consequences for antisocial involvement • consequences for prosocial involvement Bonding SKills • followed directions • parent’s problem-solving skills • child’s problem-solving skills • dyad’s problem-solving skills • parent’s recognition skills • parent’s verbal skills • child’s verbal skills • general parenting skill

39. Training of Coders • Noldus Observer software is used to code videotaped family discussions • 160-175 hours are required to reach an acceptable level of interrater agreement • Training involves extensive discussion of cultural differences in behavior and in the meaning of behavior

40. Examples • All families in the examples have provided consent to use their observations for training and research purposes outside of the specific project.

41. Tasks for Children 2-8 years • Child’s Game • Caregiver’s Game • Clean Up

42. Macro-Code: Child 2-8 Child’s Game (5 minutes): Parent is asked to play with child, following the child’s lead. • child antisocial involvement • low general parenting skill

43. Macro-Code: Child 2-8 • opportunities for prosocial involvement • child prosocial involvement • rewards for prosocial involvement Caregiver’s Game(5 minutes): Parent is asked to have the child follow what they want to play with.

44. Tasks for Children 9+ years • Expectations and Guidelines -10 minutes • Solving Family Problems – 10 minutes • Recognition- 5 minutes

45. External ConstraintsExpectations and Guidelines • Parent and Teen are Asked to Discuss rules/guidelines • Guidelines • positive • During the task, caregiver communicated clear expectations/rules/ guidelines about the child/teen’s use of (tobacco, alcohol, drugs, sexual activity, violence)

46. Skills for InvolvementRecogniton • Positive • Father was able to identify a positive attribute about the child/teen. • Family members demonstrated genuine warmth and caring toward each other.

47. Reliability and Predictive Validity: SSDP Intergenerational Project Prosocial Guidelines 0.84 -0.47** Prosocial Opportunities 0.82 -0.29+ Dyad Involvement In The Activity 0.78 -0.30* Child Prosocial Involvement 0.86 -0.24 Dyad Positive Involvement 0.92 -0.37* Consequences For Prosoc. Invol. 0.82 -0.27+ Bonding 0.82 -0.22 Followed Directions 0.64 -0.35* Parent Problem-solving Skills 0.83 -0.35* Child Problem-solving Skills 0.74 -0.32* Dyad Problem-solving Skills 0.85 -0.44** Low Skilled Parenting 0.74 0.51*** Corr. with Parent Macro-Code Scale alpha Binge Drinking