Group-Level Measurement

1. Group-Level Measurement Katherine Klein University of Pennsylvania kleink@wharton.upenn.edu CARMA Presentation February 2007

2. Why Group-Level Measurement? • Burgeoning of multilevel theory and research in last 25 years • Great progress in conceptualizing and measuring group-level constructs • Especially shared constructs • Continuing challenges and opportunities • Especially regarding configural constructs

3. A Few Terms and Assumptions • I’ll refer to groupsbut much or all of what I say will apply as well to organizations, departments, stores, etc. • I’ll focus on the creation and use of original survey measures to assess group constructs. • I’ll address statistical issues in passing only. • But see past CARMA presenters including James LeBreton, Gilad Chen, Paul Bliese, Dan Brass, Steve Borgatti, and others

4. Roadmap • Fundamentals: Theory First • Construct Types: Global, Shared, and Configural Constructs • Practicalities and Technicalities • Survey Wording • Sampling • Qualitative Groundwork • Single-source Bias • Justifying Aggregation • Opportunities and Challenges • The Configuration of Diversity • Social Network Analysis

5. Fundamentals: Theory First • Constructs are our building blocks in developing and in testing theory. • High quality measures are construct valid. • The development of construct valid measures thus begins with careful construct definition. • Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000): • Global, shared, or configural.

6. Global Constructs • Relatively objective, easily observable, descriptive group characteristics. • Originate and are manifest at the group level. • Examples: • Group function, size, or location. • No meaningful within-group variability. • Measurement is generally straightforward.

7. Shared Constructs • Group characteristics that are common to group members • Originate in group members’ attitudes, perceptions, cognitions, or behaviors • Which converge as a function of attraction, selection, socialization, leadership, shared experience, and interaction. • Within-group variability predicted to be low. • Examples: • Group climate, norms, leader style. • Measurement challenges are well understood.

8. Configural Group-Level Constructs • Group characteristics that describe the array, pattern, dispersion, or variability within a group. • Originate in group member characteristics (e.g., demographics, behaviors, personality, attitudes) • But no assumption or prediction of convergence. • Examples: • Rates, diversity, fault-lines, social networks, team mental models, team star or weakest member. • Measurement challenges are less well understood.

9. A Related Framework: Chan’s (1988) Composition Typology • Shared Constructs • Direct consensus models (e.g., group norms) • Referent shift models (e.g., team efficacy) • Configural Constructs • Dispersion model (e.g., climate strength) • Additive models (e.g., mean group member IQ) • Multilevel, Homologous Models • Process model (e.g., efficacy-performance relationship)

10. Construct Definition Complexities:An Example: Shared Leadership • Shared leadership • “A dynamic, interactive influence process among individuals in work groups in which the objective is to lead one another to the achievement of group goals… [It] involves peer, or lateral, influence and at other times involves upward or downward hierarchical influence” • Conger & Pearce, 2003, p. 286 • Is this a shared construct, or a configural construct, or … ?

11. Construct Definition Complexities:An Example: Shared Leadership • Well, how would you measure it? • Shared team leadership as a shared construct • “Team members share in the leadership of this team.” • “Many team members provide guidance and direction for other team members.” • Shared team leadership as a configural construct (network density): • “To what extent do you consider _____ an informal leader of the team?”

12. Construct Definition Complexities:An Example: Shared Leadership • Calling it a “referent shift” construct is not the answer. • Referent shift is a measurement strategy, not a construct type • Shifting the referent in an unthinking manner can be quite problematic: • The members of my team… • “Express confidence that we will achieve our goals” • “Will recommend that I am compensated more if I perform well” • “Are friendly and approachable” • “Rule with an iron hand”

13. A Quick Recap • Theory first: Define and explain the nature of your group-level constructs. • Is it a clearly objective description of the group? • If yes, a global construct. • Do you expect within-group agreement? • If yes, a shared construct. • Does it describe the group in terms of the pattern or array of group members on a common attribute? • If yes, a configural construct.

14. Now What? • Having defined your constructs, the goal is to create measures that: • Are construct valid • Show homogeneity within (shared constructs) • Show variability between (all group-level constructs) • Practicalities and technicalities • Survey wording • Sampling • Qualitative groundwork • Minimizing single-source bias • Testing for aggregation

15. Survey Wording:Global Constructs • Draw attention to objective descriptions of each group. • Gather data from experts and observers (SMEs) who can provide valid information about the groups in question. • No need to gather data from individual respondents within groups • Use language that fits your sample.

16. Survey Wording: Shared Constructs • Draw attention to shared group characteristics • Use a group referent rather than individual referent to enhance: • Within group agreement • Between group variability • Predictive validity • Gather data from individual respondents so within-group agreement can be assessed. • Actual consensus methods (discussion prior to group survey completion) work well but are labor-intensive.

17. Survey Wording:Configural Constructs • Draw attention to individual group member characteristics by using an individual referent. • Gather data from experts and observers (SMEs) who can provide valid information regarding individual group members, or gather data from individual respondents within groups. • The challenge is perhaps less in the survey wording than in operationalizing the array or pattern of interest.

18. Sampling • Substantial between-group variability is essential. Seek samples in which groups vary considerably on the constructs of interest • Whether they are global, shared, or configural. • Statistical power reflects both: • Group sample size (n of groups) • Within-group sample size • When group size is large (number of respondents per group), measures of shared constructs are more reliable. • More research needed on power in multilevel analyses.

19. Qualitative Groundwork • The survey wording and sampling guidelines seem fairly obvious and easy, but … • Check your assumptions in the field prior to survey data collection. • Are you measuring the right “groups”? • Example: Grocery stores or departments? • Is there meaningful between-group variability? • Example: Fast food chain • Are you measuring the right variables, and not too many of them? • Beware the blob.

20. Single-Source Bias • Group-level correlations between measures of shared group constructs may be disturbingly high. • Examples: • Transformational and transactional leadership • Task, emotional, and procedural conflict • Aggregation does not “average away” response biases. • Rather, group members may share response biases • Halo, logical consistency, social desirability • Response bias may be particularly influential when respondents must make subtle distinctions among constructs.

21. Single-Source Bias:Beating the Blob • Survey measures • Choose and measure truly distinct constructs • Use different survey response formats • Survey design • Keep survey items measuring distinct constructs separate. • Help respondents recognize the distinction between leadership types, or conflict types, for example.

22. Single-Source Bias:Beating the Blob • Survey analysis • Randomly split the within-group sample of respondents during data analysis. • All receive the same survey, but half provide IV and the other half provide the DV for analyses • Survey administration • Randomly split the within-group sample of respondents during data administration. • Respondents receive distinctive surveys. Half receive the IV survey and the other half receive the DV survey.

23. A Quick Recap • Having • Defined our constructs • Written our survey items • Conducted qualitative groundwork • Sampled appropriately • Taken steps to reduce single source bias • We’re almost ready for hypothesis testing • But first: We need to justify aggregation

24. Justifying Aggregation • Why is this essential? • In the case of shared constructs, our very construct definitions rest on assumptions regarding within- and between-group variability. • If our assumptions are wrong, our construct “theories,” our measures, and/or our sample are flawed and so are our conclusions. • So, test both: • Within group agreement • The construct is supposed to be shared, but is it really? • Between group variability (reliability) • Groups are expected to differ significantly, but do they really?

25. Justifying Aggregation: rwg(j) • Developed by James. Demaree, & Wolf (1984) • Assesses agreement in one group at a time. • Compares actual to expected variance. • Answers the question: • How much do members of each group agree in their responses to this item (or this scale)? • Highly negatively correlated with the within group standard deviation • Valid values range from 0 to 1 • Rule of thumb: rwg(j) of .70 or higher is acceptable

26. Justifying Aggregation: rwg • Common to report average or median rwg(j) for each group for each variable: • If rwg(j) is below .70 for one or more groups, check: • Does the group have low rwg(j) values on several variables? • Do many groups have low rwg(j) values on this variable? • Remember: rwg(j) indicates within-group agreement, not between-group variability. • Beware: When variance in a group exceeds expected variance, out of range rwg(j) result.

27. Justifying Aggregation: h2 • Assesses between-group variance relative to total variance, across the entire sample. • Based on a one-way ANOVA • Answers the question: • To what extent is variability in the measure predictable from group membership? • The F-test provides a test of significance • The larger the sample of individuals, the more likely eta2 is to be significant. • Beware: h2 may be inflated when group sizes are small (under 25 individuals per group) • But, this is an easy way to begin tests of aggregation

28. Justifying Aggregation: ICC(1) • Assesses between-group variance relative to total variance • Based on a one-way ANOVA • Answers the question: • To what extent is variability in the measure predictable from group membership? • The F-test provides a test of significance • Based on h2 but controls for the number of predictors relative to the total sample size, so ICC(1) is not biased by group size.

29. Justifying Aggregation: ICC(2) • Assesses the reliability of the group means (i.e., between-group variance) in a sample, based on ICC (1) and group size. • Answers the question: • How reliable are between-group differences on the measure? • Reflects ICC(1) and within-group sample size • Example: If ICC(1) = .20 and: • Mean group size is 5, expected ICC(2) = .56 • Mean group size is 20, expected ICC(2) = .71

30. Justifying Aggregation: An Example

31. A Quick Recap • The hope is that we have successfully: • Defined our constructs. • Written our survey items. • Conducted qualitative groundwork. • Collected data from a large sample of groups. • Taken steps to reduce single source bias. • Justified aggregation. • And moved on to test our hypotheses. • So, what remains?

32. Opportunities and Challenges:The Configuration of Diversity • Configural constructs describe the array, pattern, dispersion, or variability within a group. • The easy example is diversity • Demographic diversity • Climate strength • But even the easy example isn’t so easy: What is the definition of diversity? And how should it be measured?

33. The Configuration of Diversity • A starting definition of diversity: • The distribution of differences among the members of a group with respect to an attribute, X, such as age, ethnicity, conscientiousness, positive affect or pay. • Okay, but what’s maximum diversity? • Which team has maximum age diversity? • 20, 20, 20, 70, 70, 70 • 20, 30, 40, 50, 60, 70 • 20, 20, 20, 20, 20, 70 • 20, 70, 70, 70, 70, 70

34. The Configuration of Diversity • Diversity isn’t one thing. • It’s three things: Separation, Variety, or Disparity • The three types differ in: • Meaning or substance • Pattern or shape • Likely consequences • Appropriate operationalization • Blurring across these distinctions leads to fuzzy theory, misguided operationalizations, and potentially invalid research conclusions

35. The Configuration of DiversityExample: Three Research Teams • Team S • Members differ in their view of qualitative research. • Half of the team members respect it, half don’t. • Team V • Members differ in their discipline. • 1 psychologist, 1 sociologist, 1 anthropologist, etc. • Team D • Members differ in their rank • 1 senior professor, others are incoming graduate students.

36. Diversity as Separation • Differences in group members’ position, attitude, or opinion along a continuum • Min: Every member has the same opinion • Max: Two polarized extreme factions • Theory: Similarity-attraction • Operationalization: Standard deviation

37. Diversity as Variety • Differences in kind or category • Min: Every member is the same type • Max: Each group member is a different type • Theory: Requisite variety, cognitive resource heterogeneity • Operationalization: Blau’s index of categorical differences

38. Diversity as Disparity • Differences in concentration or proportion of valued assets or resources • Min: Every member has an equal portion of the resource • Max: One member is “rich” and all others are “impoverished” • Note: Disparity is asymmetric • Theory: Inequality, relative deprivation, tournament compensation • Operationalization: Coefficient of variation (SD/Mean)

39. The Configuration of Diversity:A Recap • Theory first • Separation is about position, attitude, or opinion • At maximum: Polarized factions • Variety is about knowledge or information. • At maximum: One of a kind • Disparity is about resources or power. • At maximum: One towers over others • Operationalize accordingly • The coefficient of variation is not a default or catch-all

40. Opportunities and Challenges: Social Network Analysis • Multilevel analysis and social network analysis have developed along separate paths. • Rich opportunities for cross-fertilization. • Social network analysis provides a means to conceptualize and operationalize configural constructs. • Illuminating the pattern or array of interpersonal ties within a group

41. Opportunities and Challenges: Social Network Analysis • Many of our shared constructs appear to rest on tacit, often fuzzy, assumptions about interpersonal ties with groups. • Examples: Cohesion, communication, coordination, knowledge sharing, shared leadership, conflict • But we know little about the configuration of interpersonal ties – the structures – that underlie our shared constructs and measures.

42. An Example: Social Network Analysis and Shared Team Conflict • When teams report high task or emotional conflict, what is the structure of interpersonal ties within the team? • As a starting point: • How dense are positive (advice) ties? • How dense are negative (difficulty) ties?

43. An Example: Social Network Analysis and Shared Team Conflict • Task and emotional conflict: The blob • r = .83 • Advice density and negative tie density: More weakly correlated • r = -.36 • Task conflict (mean task and emotional conflict), advice density, and negative tie density • Team Conflict and Advice Density: r = -.47 • Team Conflict and Difficulty Density r = .40

44. Negative Tiesin a Low Conflict Team

45. Negative Ties in a High Conflict Team

46. Advice Ties in a High Conflict Team

47. Advice Ties in a Low Conflict Team

48. Social Network Analysis:A Recap • Social network analysis illuminates the configuration of interpersonal ties in groups. • What network structures underlie our shared constructs and measures? • Do network measures provide incremental validity? • Not just density, but centralization, cliques, and more. • What explains between-group differences in network structures?

49. In Conclusion • Theory first. Define your constructs. • Are they global, shared, or configural? • Measure constructs and collect data with care • Match item wording to the construct • Conduct qualitative groundwork • Sample appropriately • Take steps to reduce single source bias • Test for aggregation • Studying configural constructs remains a challenge and an opportunity • Conceptualizing and measuring diversity • Integrating social network analysis within our arsenal

50. Some Helpful References • Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass. • Borgatti, S. P. (2003). The network paradigm in organizational research: A review and typology. Journal of Management, 29, 991-1013. • Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246. • Harrison, D. A. & Klein, K. J. (2007). What’s the difference? Diversity as separation, variety, or disparity in organizations. Academy of Management Review. • Harrison, D. A. & McLaughlin, M. E. (1996). Structural properties and psychometric qualities of organizational self-reports: Field tests of connections predicted by cognitive theory. Journal of Management, 22, 313-338. • James, Demaree, & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.