III. Assessment A. Introduction 1. Why we measure? 2. What we measure

1. III. Assessment • A. Introduction • 1. Why we measure? • 2. What we measure • 3. The case of abstract concepts • B. The OSS example • 1. Leadership • 2. The procedures • C. Modes of measurement • 1. Self-reports • a. empirical approach • b. rational approach • c. factor analysis • 2. The remaining modes • a. definitions • b. examples • D. Assessing the Assessment • 1. Reliability • a. definition • b. measurement • 2. Validity • a. criterion • b. content • c. construct

2. Assessment Why we measure? Research: to test hypotheses and facilitate classification Practical problems: personnel selection, career counseling, treatment of individuals with psychological disorders Each (more or less) requires: Specifying the outcome Identifying the relevant characteristics Developing assessment procedures Communicating results

3. Assessment What do we measure? Measure one or more attributes or characteristics (variables) Examples: Height, weight, duration of temper tantrum Reduce these attributes to numbers Different types of scales Different properties Few scales in psychology are ratio scales Abstract concepts? Agreeableness Sensation Seeking Leadership Potential

4. Measuring Concepts Process of conceptualization: working to define exactly what is meant by a given term Begin with idea Agreeableness Generate rough description Agreeableness is …. Start pointing to particular examples Multiple examples Specific examples From broad domains Specific indicators constitute the measure

5. An example: The OSS assessment Office of Strategic Services (OSS) War-time agency (WWII) Tactical and covert operations Needed to recruit and train agents Identified 7 major variables 1. Motivation 2. Energy and initiative 3. Effective intelligence 4. Emotional stability 5. Social relations 6. Leadership 7. Security

6. Measuring Leadership Leadership: ability to take the initiative in social situations, to plan and organize action, and in so doing evoke cooperation Assessment guided by principle of convergence: gather evidence from multiple viewpoints and look for agreement Multiple sources Multiple methods Used 7 different procedures

7. Important points to take from OSS assessment • First identified relevant personality variables, then designed way to assess. • Conceptualization • Then specific indicators (operationalize) • Measured each variable in different situations and with different methods • Used multiple observers. Looked for convergence. = important points about measurement • Conceptualization & operationalization • Reliability & validity

8. Methods for Measuring Personality

9. Methods for Measuring Personality Mode 1: Self-report trait inventories Typical Variables: Traits Attitudes Beliefs

10. Potential limitation of self-report methods • Misconstrue meaning • Hurry to finish or be careless • Deliberately distort answers • Lack insight into self • Try to present self favorably

11. Methods for constructing self-report measures • Empirical approaches • Rational/theoretical approaches • Factor analysis

12. Methods for Constructing Self-Reports 1. Empirical Approach: Select items that distinguish between those who possess the trait in great magnitude and those who possess the trait only in small magnitude if at all. Example: MMPI No assumptions about nature of psychiatric d/os Empirically derived

13. Empirical Approach Example: MMPI Began with 1,000 item pool taken from textbooks, interviews, and existing tests Administered these items to two groups: “Normals”: 724 visitors to U of M hospitals Groups of psychiatric patients Chose items that distinguished between “normals” and one patient group Ended up with 566 items Three validity scales and 10 clinical scales

14. Empirical Approach: MMPI Illustration of item selection

15. Empirical Approach: MMPI Criticisms: Too long Inefficient Some items are offensive Poor test-retest reliability Poor discrimination between psychiatric groups Fails to get at basic dimensions of personality Fails to assess normal range of personality

16. Methods for Constructing Self-Reports 2. Rational/Theoretical Approach: Relies on theory to determine the how and what of assessment Theory may dictate HOW to measure something Psychoanalytic theory  projective tests Projective hypothesis: If an individual is confronted with an ambiguous stimulus, the interpretation that the individual provides is to some extent a "projection" of the individual's internal characteristics.

17. Projective Tests Often contrasted with objective tests Projective tests are: Unstructured Questions asked Responses available Disguised Scored subjectively Two classic examples: Rorschach Inkblot Test Thematic Apperception Test

18. Rorschach Inkblot Test What might this be?

19. Method for Constructing Self-report • Rational/theoretical approach continued • Projective tests example of theory driving HOW to measure • Theory may also dictate WHAT to measure • Rational approach to scale construction: example

20. Rational Approach • Idea: what it is you want to measure? • Literature search: what have others done? What is similar and different? • Operationalize the construct – precise definition of variable of interest • Create initial sample pool • Adminster items

21. Rational Approach cont. 4) Create initial item pool First guiding principle: “The items of the pool should be chosen so as to sample all possible contents which might comprise the putative trait according to all known alternative theories of the trait” (Loevinger, 1957) Implications of first guiding principle: a. Pool should be broader and more comprehensive than own view b. Pool will include content that will later be shown to be irrelevant First guiding principle: “The

22. Rational Approach 4) Create initial item pool (continued) Second guiding principle: ensure that there is an adequate sample of items within each of the major content areas comprising the broadly conceptualized domain Implications of second guiding principle Create formal subscales for each content area Proportion of items dedicated to each content area should reflect importance of that area in target construct UPPS Urgency example

23. Rational Approach 7) Decide on the status of the scale Good enough  proceed with construct validation Not good enough  return to Step 4

24. Rational Approach Tips for writing good items Language: simple, straightforward, appropriate level Avoid trendy expressions Ensure variability in responding Avoid double-barreled items Decide on scale format

25. Methods for Constructing Self-Reports Factor Analytic Approach: All the characteristics found in personality test scales can be reduced to a few common factors that describe the underlying and fundamental aspects of human personality. Factor Analysis Statistical tool for data reduction Highly correlated items are measuring the same thing Scales/subscales chosen based on results

26. Example: How people cope with stress • How much did you do ______ during your most recent stressful event? Or rate each item: • 1. Took action quickly, before things could get out of hand. • 2. Refused to believe it was real. • 3. Did something concrete to make the situation better. • 4. Tried to convince myself that it wasn’t happening. • 5. Went on thinking that things were just like they were. • 6. Changed or grew as a person in a new way. • 7. Tried to look at the bright side.

27. Hypothetical correlation matrix

28. Hypothetical Factor Loadings

29. Implications • Garbage in, garbage out; brilliance in, brilliance out • Missing info. may mean missing factors E.g. Escape: smoking, drinking, eating, daydreaming about vacation, etc. • Importance of names of factors

30. Factor Analytic Approach

31. Methods for Measuring Personality Mode 1: Trait Inventories (just discussed) Mode 2: State Inventories Source: Self Time Frame: Current Function of Source: Perceives Task: Describe Your Experience Typical Variables: Preferences Judgments Feelings

32. Methods for Measuring Personality Mode 3: Ability Tests Source: Self Time Frame: Current Function of Source: Solves or decides Task: Answer Correctly Typical Variables: Knowledge Skills Abilities

33. Methods for Measuring Personality Mode 4: Informed Ratings Source: Other Time Frame: Past Function of Source: Interprets Task: Describe This Person Typical Variables: Traits Behaviors

34. Methods for Measuring Personality Mode 5: Impartial Ratings Source: Other Time Frame: Current Function of Source: Interprets or transcribes Task: Describe This Person Typical Variables: Traits Behaviors

35. Methods for Measuring Personality Mode 6: Behavioral Measures Source: Instrument Time Frame: Current Function of Source: Records Mechanically Task: Record Behavioral Response Typical Variables: Reaction times Responses

36. Methods for Measuring Personality Mode 7: Physiological Measures Source: Instrument Time Frame: Current Function of Source: Records Mechanically Task: Record Physiological Activity Typical Variables: Cortical Arousal Autonomic Arousal Sexual Arousal