Scenario-based scales: Integrating Guttman facet theory and Rasch measurement

1. Scenario-based scales: Integrating Guttman facet theory and Rasch measurement Larry H. Ludlow, Ph.D. Boston College Lynch School of Education Educational Research, Measurement, and Evaluation ludlow@bc.edu http://www.bc.edu/content/bc/schools/lsoe/academics/departments/erme.html Prepared for: Institute of Education Higher School of Economics Moscow, Russia October, 2014

2. Variable Map for AfL Did this work roughly enough as intended? Levels of Practice Peer-and Self-Assessment Learning Intentions and Success Criteria Feedback Questioning and Classroom Discussion BUT what is the “meaning” for a given teacher?

3. Average |P=50% 1 |P=50% at Average|P=50% 5 Item Response | | on each Item | 6 + + + Embedded | | | | | | | | | + + + | | | | | | | | | PSA5 + + + | | | | | | PSA1 PSA2 . | | | | | | PSA4 . + + + LISC5 PSA3 | | | FB5 FB4 . | | | QCD3 .## | | | LISC4 .## | | | FB2 LISC1 QCD5 5 .#### + + + FB3 Established ###### | | PSA5 | QCD4 .###### | | | FB1 LISC2 .######## | | | QCD1 ########### | | PSA1 PSA2 | 4 .########### + + + LISC3 Emerging .########## | | PSA4 | .########## | | LISC5 PSA3 | QCD4 ########### | | FB5 FB4 | .###### | | QCD3 | ### + + LISC4 + 3 .## | | FB2 LISC1 QCD5 Sporadic .# | | FB3 | . | | QCD4 | . | | FB1 LISC2 | . + + QCD1 + | | | | | LISC3 | | | | | | QCD4 | 2 + PSA5 + + Never | | | | | | | PSA1 PSA2 | | | | | + PSA4 + + | LISC5 PSA3 | | | FB5 FB4 | | | QCD3 | | | LISC4 | | + FB2 LISC1 QCD5 + + | FB3 | | | QCD4 | | | FB1 LISC2 | | | QCD1 | | + + + | LISC3 | | | | | | QCD4 | | | | | + + + | | | | | | + + + | | | | | | 1 | | | Don’t understand This is the level to which we would like to see them advance—50% prob of a 6. Now we can show “average” response levels showing how teachers responded. Most of the teachers are at these two levels. A few teachers are at this level. Fortunately, no one is at this level—50% prob of a “1”.

4. Teachers’ AfL Benchmark Descriptions Very useful description of how to interpret average scores. Why not make a “scenario description” exactly like we have tried to do here? Ludlow-ERME

5. Personal Engagement • According to Kahn (1990): • Personal engagement represents “the simultaneous employment and expression of a person’s ‘preferred self’ in ways that promote deep connections to one’s work (whatever form that might take, paid or unpaid) and to other people. • Such engagement leads to active, full, and satisfying involvements rather than obligatory, passive, or emotionally anemic ones. • We become personally engaged when we find that it is meaningful and safe to express our full selves and when we are psychologically ready to do so. (p.700).

6. Our Broader Definition of Engagement • A positive, enthusiastic, and affective connection with a role that both motivates individuals to invest their valuable resources and simultaneously energizes them. • Ranges across lower levels through higher levels. • Characterized by persistence, energy, focus, and interest experienced while enacting a role. • Highly correlated and range from lower to higher levels. • A unidimensionalconstruct that cuts across different adult roles we may assume.

7. What is psychological engagement? • • What does the concept of “full” engagement or “less than full” engagement look and feel like? • • Is it possible to be “disengaged”? • • Is this construct on a continuum? And, if so what is on each end of that continuum? • • How do you measure engagement? • • Can it be measured across roles or is this a role-specific concept?

8. Conceptual Model

9. Productive Engagement in Later Life

10. Sample • Conducted in September 2012 through Survey Sampling International (SSI). • Stratification for activity involvement resulted in equal numbers of respondents per activity category: • 120 paid workers • 120 volunteers • 120 caregivers • 120 informal helpers • 480 individuals age 50 to 89 • Mean age 63.18 (SD:8.32) • 60.6% Female • 38.8% Bachelors degree or higher • 90.9% Caucasian • 57.9% Married/cohabiting • 92.1% Live independently

11. Classical Test Theory (CTT) vs Item Response Theory (IRT) Classical Test Theory Item Response Theory “Modern” approach since 1980’s Based on probability notion of “expected response” for a person to an item Emphasis on meaning and interpretation of individual items Many differentstatistical models for determining “expected responses” A particularly powerful model for item/scale development is the Rasch measurement model. • “Traditional” approach since 1904 • Based on notion of “true score” • X = T + E • Emphasis is on use and application of “total score” • Typical procedures are Cronbach alpha, test-retest reliability, factor analysis • Simple items, often using Likert scoring—SA to SD

12. Rasch Model Measurement Principles • When measuring a construct, items should define a construct with these characteristics: • Variation: wide range of beliefs, attitudes, opinions, abilities • Unidimensional: address a single, common attribute of the construct • Hierarchical: progress from “easier-to-difficult” to endorse or accomplish items/tasks • Continuum: form a uniform continuum across this hierarchical, unidimensional variation

13. Ludlow-ERME High Engagement Structure of a “variable” • Every variable stretches across a continuum: levels of knowledge, cognitive abilities, affective characteristics (e.g. engagement) • We construct a deliberate hierarchical sequence of items, like steps on a ladder, that show where a person is located and what can then be expected to advance along that continuum (or up the ladder). Low Engagement

14. How to turn Rasch principles into scenarios? • Facet Theory (L Guttman) • Methodological approach to reveal how we form beliefs, attitudes, perceptions such as “engagement”. • Each such perception is based on many influences, circumstances, factors, dimensions, considerations. • Each of these influences is called a “facet”. • Each facet is made up of different levels/elements. • In essence, FT or facet design, reveals the facets and their elements.

15. Sentence mapping (Guttman) • A visual, grammatical device to link the elements of the facets. • All possible combinations of one element from each facet are formed. • We have 4 facets, each with 3 levels for a total of 3 x 3 x 3 x 3 = 81 combinations • These combinations are used to form sentences. • These sentences are paired to become our scenarios. • Our facets and elements are:

16. Choosing Scenarios • All 81 combinations are impractical, how do you choose? • Randomly select from the 81 combinations of Ii,Fi,Ei,Pi. • Extreme groups contrast procedure (B Bloom). • We chose the following combinations • 3 from the extreme positive (I1, F1, E1, P1) group, • 3 from the extreme negative (I3, F3, E3, P3), and; • 3 from the middle(I2, F2, E2, P2).

17. How did we start thinking about the enagagement construct?

18. How did we think about the “level” of the scenarios?

19. Scenarios (using Work as the “Role”) • On a typical day , are you… • Much more engaged than X • More engaged than X • About as engaged as X • Less engaged than X • Much less engaged than X • 9) Anan identifies strongly with his work (in a positive way) and sometimes gets so wrapped up in what he is doing that it is difficult to tear himself away. He gives all of himself to his work and finds that he gets energized from doing so.   • 7) Melissa is fascinated by her work and is usually intensely focused on whatever she’s working on. She is willing to invest much energy in order to do a good job and she persists when difficulties arise. • 1) Tina feels enthusiasticabout her work and is pretty focused on the task at hand. She goes above and beyond what is required and when challenges arise, she deals with them. • 6) Clair is interested in her work and pays attention to whatever she’s working on. She does what it takes and handles difficulties when they arise, though her mind wanders occasionally. • 2) Stan is somewhat interested in his work and generally focuses on whatever he’s working on. He does what is required and keeps at it when difficulties arise. • 4) Elyssafeels indifferentabout her work and often thinks about other things. She has little desire to do more than is required and has to force herself to keep going when things get difficult. • 8) Jackie is tiredof her work and usually thinks about how much she would rather be doing something else. She does not invest much energy in what she does and doesn’t go out of her way when difficulties arise. • 5) Larry feels unenthusiastic about his work and is usually thinking about other things. He does not go out of his way to get tasks done and tends to give up when any effort is required. • 3) Jamie does not care about his work and is completely checked out. He does not invest any energy at all in what he does and almost always gives up when effort is required.

20. “Variable Map” Work Harder scenarios 9) Identifies strongly with “X” (in a positive way)…gets so wrapped up it is difficult to tear self away…gives all of self to and gets energized from doing so. 7) Fascinated and intensely focused…willing to invest much energy…persists when difficulties arise. 1) Enthusiastic and pretty focused…goes above and beyond and deals with challenges 2) Somewhat interested and generally focused…does what is required and keeps at it when difficulties arise. 6) Interested and pays attention…does what it takes and handles difficulties, though mind wanders occasionally. 4) Feels indifferent and thinks about other things…little desire to do more than is required…forces self to keep going when things difficult. 8) Tired and usually thinks about how much rather be doing something else…does not invest much energy and doesn’t go out of way when difficulties arise. 5) Unenthusiastic and usually thinking about other things…does not go out of way…gives up when effort is required. 3) Does not care…completely checked out…does not invest any energy…gives up Higher level of engagement • MEASURE Person - MAP - Item • <more>|<rare> • 8 # + • | • | • | • | • 7 + • | • | • | • | • 6 + • | • | • | • | • 5 + • | • | • | • | • 4 + • .# | • | • .## | • # | • 3 + • ## | • | • ########## | SC9 • ######## | • 2 + • .########## | • # M| SC1 SC7 • ##### | • .### | • 1 .## + • | • | • ### | SC2 • .## | SC6 • 0 .# +M • # | • | • . | • . | • -1 . + • | SC4 SC8 • | • .# | • | SC5 • -2 . + SC3 • | • | • | • | • -3 + • <less>|<frequent> • EACH "#" IS 2. EACH "." IS 1. High scoring people Low scoring people Lower level of engagement Easier scenarios

21. Work • MEASURE Person - MAP - Item • <more>|<rare> • 8 # + • | • | • | • | • 7 + • | • | • | • | • 6 + • | • | • | • | • 5 + • | • | • | • | • 4=41 + • .# | • | • .## | • # | • 3=39 + • ## | • | • ########## | SC9 • ######## | • 2=35 + • .########## | • # M| SC1 SC7 • ##### | • .### | • 1=31 .## + • | • | • ### | SC2 • .## | SC6 • 0=27 .# +M • # | • | • . | • . | • -1=22. + • | SC4 SC8 • | • .# | • | SC5 • -2=19 . + SC3 • | • | • | • | • -3 + • <less>|<frequent> • EACH "#" IS 2. EACH "." IS 1. But what does a score actually mean? For example, what is the difference between a person who scored 38 and one who scored 22? Where are those scores and what is their “engagement difference? 9) Identifies strongly with “X” (in a positive way)…gets so wrapped up it is difficult to tear self away…gives all of self to and gets energized from doing so. 4) Feels indifferent and thinks about other things…little desire to do more than is required…forces self to keep going when things difficult. 8) Tired and usually thinks about how much rather be doing something else…does not invest much energy and doesn’t go out of way when difficulties arise.

22. Person - MAP - Item CAREGIVING + | | | | + | | | | + | | | | + . | | . | | + . | | | | # + ## | | .### | ##### | SC9 .##### + . | ##### | SC7 .#### | SC1 .#### M| .#### + ##### | ## | | SC6 ## | SC2 ##### +M . | .# | # | . | + . | SC4 SC8 .# | # | SC3 SC5 | # + . | | . | | + EACH "#" IS 2. EACH "." IS 1. Person - MAP - Item INFORMAL HELPING + | | | | + . | | | | + | | . | | . + | | | | # + | ######### | | SC9 | .#### + | ####### | | SC1 SC7 .####### | M+ .###### | ### | | ##### | + .## | SC2 ### | | SC6 .# | #### +M . | | .# | . | + | | . | . | + SC8 . | SC4 | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1. Person - MAP - Item VOLUNTEER .## + | | | | + | | | | . + | | | | ### + | .## | | | .## + | ######### | | SC9 ########## | . + ###### | M| | SC1 SC7 .###### | .### + | .## | . | .### | .# + | SC2 .## | . | SC6 | # +M | | . | | + | | . | | + SC8 | SC4 . | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1. How well did our predicted scenario order come out? • MEASURE Person - MAP - Item • WORK • 8 # + • | • | • | • | • 7 + • | • | • | • | • 6 + • | • | • | • | • 5 + • | • | • | • | • 4 + • .# | • | • .## | • # | • 3 + • ## | • | • ########## | SC9 • ######## | • 2 + • .########## | • # M| SC1 SC7 • ##### | • .### | • 1 .## + • | • | • ### | SC2 • .## | SC6 • 0 .# +M • # | • | • . | • . | • -1 . + • | SC4 SC8 • | • .# | • | SC5 • -2 . + SC3 • | • | • | • | • -3 + • EACH "#" IS 2. EACH "." IS 1. G A P G A P G A P G A P G A P G A P G A P G A P

23. Person - MAP - Item CAREGIVING + | | | | + | | | | + | | | | + . | | . | | + . | | | | # + ## | | .### | ##### | SC9 .##### + . | ##### | SC7 .#### | SC1 .#### M| .#### + ##### | ## | | SC6 ## | SC2 ##### +M . | .# | # | . | + . | SC4 SC8 .# | # | SC3 SC5 | # + . | | . | | + EACH "#" IS 2. EACH "." IS 1. Person - MAP - Item INFORMAL HELPING + | | | | + . | | | | + | | . | | . + | | | | # + | ######### | | SC9 | .#### + | ####### | | SC1 SC7 .####### | M+ .###### | ### | | ##### | + .## | SC2 ### | | SC6 .# | #### +M . | | .# | . | + | | . | . | + SC8 . | SC4 | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1. Person - MAP - Item VOLUNTEER .## + | | | | + | | | | . + | | | | ### + | .## | | | .## + | ######### | | SC9 ########## | . + ###### | M| | SC1 SC7 .###### | .### + | .## | . | .### | .# + | SC2 .## | . | SC6 | # +M | | . | | + | | . | | + SC8 | SC4 . | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1. Where is the average level of engagement? • MEASURE Person - MAP - Item • WORK • 8 # + • | • | • | • | • 7 + • | • | • | • | • 6 + • | • | • | • | • 5 + • | • | • | • | • 4 + • .# | • | • .## | • # | • 3 + • ## | • | • ########## | SC9 • ######## | • 2 + • .########## | • # M| SC1 SC7 • ##### | • .### | • 1 .## + • | • | • ### | SC2 • .## | SC6 • 0 .# +M • # | • | • . | • . | • -1 . + • | SC4 SC8 • | • .# | • | SC5 • -2 . + SC3 • | • | • | • | • -3 + • EACH "#" IS 2. EACH "." IS 1.

24. Person - MAP - Item CAREGIVING + | | | | + | | | | + | | | | + . | | . | | + . | | | | # + ## | | .### | ##### | SC9 .##### + . | ##### | SC7 .#### | SC1 .#### M| .#### + ##### | ## | | SC6 ## | SC2 ##### +M . | .# | # | . | + . | SC4 SC8 .# | # | SC3 SC5 | # + . | | . | | + EACH "#" IS 2. EACH "." IS 1. Person - MAP - Item INFORMAL HELPING + | | | | + . | | | | + | | . | | . + | | | | # + | ######### | | SC9 | .#### + | ####### | | SC1 SC7 .####### | M+ .###### | ### | | ##### | + .## | SC2 ### | | SC6 .# | #### +M . | | .# | . | + | | . | . | + SC8 . | SC4 | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1. Person - MAP - Item VOLUNTEER .## + | | | | + | | | | . + | | | | ### + | .## | | | .## + | ######### | | SC9 ########## | . + ###### | M| | SC1 SC7 .###### | .### + | .## | . | .### | .# + | SC2 .## | . | SC6 | # +M | | . | | + | | . | | + SC8 | SC4 . | | SC3 SC5 | + EACH "#" IS 2. EACH "." IS 1. Where is the same score across the 4 roles? • MEASURE Person - MAP - Item • WORK • 8 # + • | • | • | • | • 7 + • | • | • | • | • 6 + • | • | • | • | • 5 + • | • | • | • | • 4 + • .# | • | • .## | • # | • 3 + • ## | • | • ########## | SC9 • ######## | • 2 + • .########## | • # M| SC1 SC7 • ##### | • .### | • 1 .## + • | • | • ### | SC2 • .## | SC6 • 0 .# +M • # | • | • . | • . | • -1 . + • | SC4 SC8 • | • .# | • | SC5 • -2 . + SC3 • | • | • | • | • -3 + • EACH "#" IS 2. EACH "." IS 1. 38 38 38 38

25. Was it possible to offer a new definition of engagement? • Yes! • Was it possible to construct meaningful and useful scenario items that captured one’s level of engagement? • Yes! • Was it possible to define engagement and construct scenario-based scales that were invariant across the adult roles of work, caregiving, informal helping and volunteering? • Yes! Conclusions & Implications

27. Patterns of Practice for Equity (POPE) The literature clearly identified the importance of teachers: • Selecting worthwhile content and designing learning opportunities aligned to valued outcomes • Connecting to students’ lives and experiences • Creating learning-focused, respectful and supportive learning environments • Taking an inquiry stance through using evidence to scaffold learning, and improve teaching, • Taking responsibility for further professional engagement and learning, and • Challenging inequities.  

28. Teaching for Equity Enactment Scenarios We potentially have six interconnected components (aspects/facets) of patterns of practice for equity but this number may be modified. We will initially construct scenario-type items for each selected pattern of practice for equity facet, defining enactment of that dimension from excellent, through moderate, to limited (high, medium, low). We will use the following steps to develop and test the Rasch tool: • Select the patterns of practice for equity facets to be included in the tool and identify the elements of each facet. Typically the identification of facets and their elements requires considerable iterations and revisions. • Select an exemplar term, plus synonyms for this term, for each pattern of enactment (excellent, moderate, limited) of the selected facets. • Construct mapping sentences. These contain the formal elements of the facets and the informal components of natural language in order to contextualize the facets. • Determine which patterns of enactment will form part of the tool. We will combine the exemplar terms from one scale with those of the other facets. If we select 6 facets, each with three levels, there are potentially 36 = 729 possible combinations. We will, instead, use an “extreme groups contrast” procedure with three scenarios from the higher facet level, three from the moderate level and three from the lower level. We intend to follow this approach as it maximizes the possibility of working with scenarios that capture the boundaries of patterns of practice for equity enactment levels.