1 / 81

Measuring Group-Level Psychological Properties (A Tribute to Larry James)

Measuring Group-Level Psychological Properties (A Tribute to Larry James). Daniel A. Newman University of Illinois. Daniel A. Newman, Ph.D. Overview. Group-Level Psychological Properties ? Psychological Climate Group-Level vs. Individual-Level Constructs Aggregation Bias

napua
Download Presentation

Measuring Group-Level Psychological Properties (A Tribute to Larry James)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Group-Level Psychological Properties(A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D.

  2. Overview • Group-Level Psychological Properties? • Psychological Climate • Group-Level vs. Individual-Level Constructs • Aggregation Bias • Why we need rWG(Within-group agreement) • Justifying Aggregation • rWG(J) for multi-item scales • Agreement vs. Reliability 2

  3. Overview • Group-Level Psychological Properties? • Psychological Climate • James & Jones (1974), Jones & James (1979), James & Sells (1981), James (1982), James et al. (1988), James & James (1989) • Aggregation Bias James (1982), James et al. (1980) • Why we need rWG(Within-group agreement) • James (1982), James, Demaree, & Wolf (1984; 1993), George & James (1993) • rWG(J) for multi-item scales • James, Demaree, & Wolf (1984), LeBreton, James, & Lindell (2005) 3 3

  4. Overview • Group-Level Psychological Properties? • Psychological Climate • James & Jones (1974), Jones & James (1979), James & Sells (1981), James (1982), James et al. (1988), James & James (1989) • Aggregation Bias James (1982), James et al. (1980) • Why we need rWG(Within-group agreement) • James (1982), James, Demaree, & Wolf (1984; 1993), George & James (1993) • rWG(J) for multi-item scales • James, Demaree, & Wolf (1984), LeBreton, James, & Lindell (2005) 4 4 4

  5. Quotes & Equations • In summarizing Larry James’s contributions to Multilevel Theory, I’ll use a two-pronged approach: • Quotes • Equations 5 5

  6. Quotes & Equations • In summarizing Larry James’s contributions to Multilevel Theory, I’ll use a two-pronged approach: • Quotes • Equations 6

  7. Levels of Analysis • In social science, hypothetical constructs reside at multiple levels of analysis (or levels of aggregation): • National Level: Culture • Organizational Level: Organizational Climate, CEO personality, Strategy • Team Level: Team efficacy, Norms, Leader style • Individual Level: Attitude, Personality, Job Performance, Psychological Climate

  8. Levels of Analysis Organizational Group Individual

  9. Levels of Analysis • Individuals are nested within Groups • Groups are nested within Organizations • One level can influence another • Group norms influence individual behavior • Individual behaviors aggregate to produce group/team performance

  10. Psychological Climate • Psychological Climate – ‘the meaning an individual attaches to a work environment’ • Organizational Climate – ‘the aggregated meaning; i.e., the typical, average, or usual way people in a setting [work environment] describe it’ • Schneider (1981, pp. 4-5), as cited by James (1982) 10

  11. Psychological Climate • Psychological Climate – individual level construct • Organizational Climate – group level construct 11 11

  12. Psychological Climate • “… perceptual agreement implies a shared assignment of psychological meaning, from which it follows that an aggregate (mean) climate score provides the opportunity to describe an environment in psychological terms.” • “Furthermore, given perceptual agreement, I submit that a climate construct at the aggregate level is defined in precisely the same manner as it is at the individual level.” • James (1982, p. 221) 12 12

  13. Psychological Climate • Relationship between organizational climate and psychological climate: • PC = psychological climate perception of person in a group • OC = organizational climate of the group 13 13 13 13

  14. Psychological Climate • Relationship between organizational climate and psychological climate: • PCpg = psychological climate perception of person p in group g • OC0g = organizational climate in group g 14 14

  15. Psychological Climate • Relationship between organizational climate and psychological climate: • PCpg = psychological climate perception of person p in group g • OC0g = organizational climate in group g • upg = deviation of person p’s individual psych. climate perception from group g’s org. climate 15

  16. Psychological Climate • James & Jones (1974), reviewed 3 approaches to conceptualize & measure org. climate: • Org.-Level Attribute, Multiple Measures • Org.-Level Attribute, Perceptual Measures • Indiv.-Level Attribute, Perceptual Measures* • * Introduced the term, “Psychological Climate” 16 16

  17. James & Jones (1974) • “Returning to the perceptual definition of organizational climate, it would seem that the reliance on perceptual measurement may be interpreted as meaning that organizational climate includes not only descriptions of situational characteristics, but also individual differences in perceptionsand attitudes. This is somewhat confusing if one wishes to employ organizational climate as an organizational attribute or main effect, since the use of perceptual measurement introduces variance which is a function of differences between individuals and is not necessarily descriptive of organizations or situations. Therefore, the accuracy and/or consensus of perceptionmust be verified if accumulated perceptual organizational climate measures are used to describe organizational attributes (Guion, 1973).” (p. 1103) 17 17

  18. Jones & James (1979) • “The [conceptual] argument for aggregating perceptually based climate scores (i.e., psychological climate scores) appears to rest heavily on three basic assumptions: first, that psychological climate scores describe perceived situations; second, that individuals exposed to the same set of situational conditions will describe these conditions in similar ways; and third, that aggregation will emphasize perceptual similarities and minimize individual differences. Based on this logic, it is generally presumed that empirically demonstrated agreement among different perceivers implies that these perceivers have experienced common situational conditions (Guion, 1973; Insel & Moos, 1974; James & Jones, 1974; Schneider, 1975a),” • (p. 206). 18 18 18

  19. James & Jones (1974) • “Although this school of thought [from Schneider and others] assumes that situational and individual characteristics interact to produce a third set of perceptual, intervening variables, such an assumption does not mean that perceived climate is different from an individual attribute. Rather, the intervening variables are individual attributes which provide a bridge between the situation and behavior.” • (p. 1107) • So … “Psychological Climate” is born! 19 19 19

  20. James (1982) • “current thinking in climate suggests that the unit of theory for climate, including organizational climate, is the individual, and the appropriate unit to select for observation is the individual. This thinking is based on the view that climate involves a set of macro perceptions that reflect how environments are cognitively represented in terms of their psychological meaning and significance to the individual.” • (p. 219) • So … measuring organizational climate (an org.-level attribute) involves an individual-level true score (i.e., psychological climate). 20 20 20 20

  21. James et al. (1988) • “Shared assignment of meaningjustifies aggregation to a higher level of analysis (e.g., groups, subsystems, organizations) because it furnishes a way of relating a construct (PC) that is defined and operationalized at one level of analysis (the individual) to another form of the construct at a different level of analysis (e.g., group climate, subsystem climate, OC). Although the unit of analysis for the aggregate psychological variable is the situation (e.g., group, subsystem, organization), the definition and basic unit of theory remains psychological.” • (p. 130, from Organizations Do Not Cognize) 21 21 21 21 21

  22. James & James (1989) General PC .85 .81 .77 .86 Group Warmth & Cooperat. Leader Support Role Stress, Conflict, Ambiguity Job Autonomy, Challenge • PC = Cognitive evaluation of work environment • See James & Sells (1981), Jones & James (1979) 22 22 22 22 22

  23. Psychological Climate Psych. Climate Job Satisfaction/Affect • Reciprocal relationship between PC and Job Satis./Affect • James & Tetrick (1986), James & James (1992) 23 23 23 23 23 23

  24. Psychological Climate • Summary: • There is a group-level organizational reality (“the situation”) • That reality is reflected in individual-level, psychological perceptions • The individual-level psychological climate perceptions are a meaningful locus of theory • The individual perceptions can be aggregated to represent a group-level, psychological property [if perceptions are shared] 24 24

  25. Aggregation Bias • Aggregation – combining micro-level data so it can represent the macro-level (typically, by taking an average of micro-level responses) • The aggregate of individuals’ scores represents the group-level construct

  26. Levels of Analysis Organizational Group Individual 26

  27. Aggregation • Ecological fallacy – generalizing group-level (aggregate) results to the individual level • Because we know group collectivism is related to group-level cooperation, we inaccurately assume individual collectivism is related to individual cooperativeness. • Atomistic fallacy – generalizing individual-level results to the group (aggregate) level • Because we know indiv. IQ is strongly related to indiv.-level job performance, we inaccurately assume group IQ is strongly related to group performance.

  28. Aggregation • The Truth about Aggregates: • If the individual-level correlation between X and Y is rindiv. = .3, this does not imply that the group-level correlation between X and Y is rgroup = .3. • Likewise, if the group-level correlation between X and Y is rgroup = .3, this does not imply that the individual-level correlation between X and Y is rindiv. = .3.

  29. Aggregation Direction of a correlation (+ or -) can change when we move from the individual level to the group level. Within-Group Correlation Between-Group Correlation Y X

  30. Aggregation Example) Foreign birth & Illiteracy (Robinson, 1950). rindiv. = .12; rgroup(states) = -.53 Within-Group Correlation Between-Group Correlation Y X

  31. Aggregation Total correlation is a combination of the individual-level correlation and the group-level correlation. Within-Group Correlation Between-Group Correlation rtotal rwithin Y rbetween Total Correlation X

  32. Aggregation • Total correlation is a combination of the individual-level (within) correlation and the group-level (between) correlation.

  33. Aggregation • Specifically, • rtotal = overall X-Y correlation, ignoring • group membership • rbetween = between-groups X-Y correlation • rwithin = within-groups X-Y correlation • (from ANOVA; DV= X, IV= group) • [like R2; variance in X accounted for by group membership, then inflated by the unreliability of group means; i.e., .]

  34. Aggregation • For example, suppose • rbetween = -.45 = between-groups X-Y correlation • rwithin = .20 = within-groups X-Y correlation • = .64 (from ANOVA; DV= X, IV= group) • = .81 (from ANOVA; DV= Y, IV= group) • Then …

  35. Aggregation • For example, suppose • rbetween = -.45 = between-groups X-Y correlation • rwithin = .20 = within-groups X-Y correlation • = .64 (from ANOVA; DV= X, IV= group) • = .81 (from ANOVA; DV= Y, IV= group) • Then …

  36. Aggregation Total correlation is a combination of the individual-level correlation and the group-level correlation. Within-Group Correlation Between-Group Correlation rtotal rwithin Y rbetween Total Correlation X

  37. Aggregation • Implications: • Even if total correlation between X and Y (rtotal) is statistically significant, • rwithin might not be • rbetweenmight not be • * Many studies in top journals report total relationships between variables, while ignoring nesting/ nonindependence (e.g., different groups, different jobs, different supervisors). Consideringlevels of analysis could potentially change the results!

  38. Aggregation • Implications: • So-called “aggregation bias” – when rbetween is larger than rtotal • Only occurs if rbetweenhappens to be larger thanrwithin 38

  39. Aggregation Bias • Implications: • Don’t look at rtotal to draw inferences about rwithin! • Don’t look at rtotal to draw inferences about rbetween! • See James (1982) and James, Demaree, & Hater (1980), who applied similar formulae to estimate bias in both h2 and corr.’s between aggregated situational (OC) and individual difference variables.

  40. Aggregation Bias • Summary: • When we aggregateindividual-level measures (e.g., psychological climate) to represent organizational attributes (e.g., organizational climate), then all the theoretical and empirical relationships can change. • Aggregation of the same measures can create a different construct! 40 40 40

  41. Why We Need rWG • Justifying Aggregation • “… organizational climate is the overall meaning derived from the aggregation of individual perceptions of a work environment (i.e., the typical or average way people in an organization ascribe meaning to that organization) (James, 1982; Schneider, 1981). Thus, organizational climate can be viewed as the outcome of aggregating individuals’ psychological climates. The important caveat is that these psychological climates are shared in order to make the inference that an organizational climate exists.” • James et al. (2008, pp. 15-16) 41 41

  42. Why We Need rWG • Group-Level Consensus Constructs • In measuring group consensus constructs, agreement and reliability are tools used to justify aggregation of individual-level responses to the group level • Agreement and reliability help us gauge how well the average across individual responses represents the group.

  43. Why We Need rWG Group-Level Consensus Constructs Organizational Climate (average) Psych. Climate, Person #1 Psych. Climate, Person #2 Psych. Climate, Person #3

  44. Why We Need rWG • Overview • Aggregation/Composition Models • Chan (1998) • Kozlowski & Klein (2000) • Agreement • rWG family of indices • Reliability • ICC(1) • ICC(2) See Bliese, 2000

  45. Why We Need rWG • Aggregation/Composition Models • Chan (1998) • Kozlowski & Klein (2000) • Both typologies include consensus models • Use the mean of individual responses to represent the group-level construct • Assume isomorphism(James, 1982) • Require high within-group agreement

  46. Why We Need rWG • Within-Group Agreement – degree to which ratings from individuals are interchangeable • Agreement-based tests reflect degree to which raters provide essentially the same rating • Three dominant indices designed to assess within-group agreement: • James et al.’s (1984) rWG(J) • Lindell et al.’s (1999) • Burke, Finkelstein, & Dusig’s (1999) AD index

  47. George & James (1993) • “The key statistical test of the appropriateness of aggregation to the group level of analysis is that there is within-group agreement on the variable in question. If there is agreement within groups on the theorized group-level variable, then the aggregate may be used in subsequent analyses.” • … agreement within a group is not conditional on between-groups differences. For example, in a scenario that Yammarino and Markham portray, in which all members in each group have the same moderately high score, both agreement and aggregation may be justified provided that aggregation to the group level was theoretically based. However, there would be no group effect inasmuch as the group means do not vary under these conditions.” • (p. 799) 47 47 47 47 47 47 47

  48. Why We Need rWG • Within-Group Agreement • For single items: • = observed variance of single item • = theoretical null variance (represents “zero agreement”) • rWG = “1 - observed variance over expected variance”

  49. Why We Need rWG • Summary: • Under consensus composition models (with isomorphism across levels), within-group agreement is needed to justify aggregation. • Within-group agreementis even more essential than ICC(1) and ICC(2), both of which depend upon between-group variance. • Within-group agreement = shared psychological meaning! • rWG is the key to measuring group-level psychological properties! 49 49 49 49

  50. rWG(J) for Multi-Item Scales • rWG(J) is NOT the same as rWG! • rWG = for single items • rWG(J) = for multiple-item climate scale 50 50

More Related