Perspectival Diversity and Consensus Analysis

1. Perspectival Diversity and Consensus Analysis John Gatewood . . . . . . Lehigh University John Lowe . . . . . . . Cultural Analysis Group AAA Meetings, Philadelphia, Dec 5, 2009

2. Preview • INTRODUCTORY REMARKS • Problem of “culture-sharing” (and non-sharing) • Basic patterns of inter-informant agreement … adding“perspectival diversity” to the list • OUR CURRENT STUDY • Assessing effects of different distributional patterns on consensus analysis’s key indicators • Some initial findings • CONCLUSIONS • Methodological “lessons” for researchers • Future directions

3. INTRODUCTORY REMARKS

4. Problem of “Culture-Sharing” • By definition, culture is socially transmitted knowledge; hence, it must be “shared” … but sharing is always a matter of degree • Hence, two related issues for any given cultural domain: • How much knowledge is shared? (the AVERAGE “cultural competence”) • How is the knowledge socially distributed? (the DISTRIBUTIONAL PATTERN) • KEY INSIGHT = assess degree of culture-sharing by examining patterning of inter-informant agreement

5. Basic Patterns of Agreement • Boster (1980, 1985) … four basic patterns of agreement (paraphrasing & expanding): • UNIFORM agreement – traditional view of culture • RANDOM agreement – free variation → no culture • EXPERTISE gradient – experts tend to agree with one another whereas non-experts deviate randomly • SUBCULTURAL variation – more than one ‘school of thought’ • Competing answer sets – different groups, different truths • Complementary knowledge – different groups systematically know different things • Romney, Weller & Batchelder (1986) … cultural consensus theory & consensus analysis

6. Perspectival Diversity … a 5th Pattern • Pilot study of credit union employees (Gatewood & Lowe 2006) No consensus in sample * + No identifiable subcultural groups => Fish-scale overlappings of partial knowledge … “perspectival diversity” i.e., social interaction and knowledge among the employees was rather departmentalized … their understandings of ‘credit unions’ reflected what they needed to know to perform their own jobs, not necessarily what might be relevant to other people _______________________ * Pilot study’s conclusion about “no consensus” turned out to be an artifact of our failure to counter-balance items in the questionnaire form (see Gatewood & Lowe 2008) … but that’s another story

7. To generalize, perspectival diversity occurs when… • All individuals have limited knowledge with respect to a given domain and to approximately the same degree • Each individual’s range of knowledge only partially overlaps with the ranges known by others And, consistent with this definition, different geometries of perspectival diversity are possible …e.g., circular pattern, linear pattern, taxonomic-hierarchical, overlapping polygons on a surface, etc.

8. OUR CURRENT STUDY

9. RESEARCH QUESTION: • Ceteris paribus, do the different distributional patterns affect the key indicators of consensus analysis? • As the average knowledge in a sample varies, do different distributional patterns “show” consensus more readily than other patterns? • Do some distributional patterns “mask” cultural consensus when other patterns “reveal” it? • If NO  … nothing to worry about [ yippee! ] • If YES  distributional patterning has an independent effect that needs to be taken into account when interpreting results of consensus analyses

10. ITEM FORMAT: • Counter-balanced Likert-style questions, i.e., 6-point “strongly agree” to “strongly disagree” response scale … because these are so common in survey research ANALYSES: • Such data can be analyzed two ways: • INFORMAL MODEL of consensus analysis … i.e., input to factor analysis is a Resp x Resp correlation matrix (data treated as interval-scale) • FORMAL MODEL of consensus analysis … i.e., input to factor analysis is a chance-corrected agreement matrix (data treated as nominal-scale, e.g., dichotomized responses)

11. Research Design … with average knowledge and distributional pattern as manipulated variables

12. “Theoretical” Predictions

13. Implementation • How to “experimentally manipulate” key parameters for different distributional models while holding others constant ?? … computer simulation to the rescue ! See: Excel “data-generating” file& Excel “findings” file

14. SOME INITIAL FINDINGS

26. Key Findings • Distributional pattern has an independent effect with respect to consensus indicators • w/r/to RATIO OF EIGENVALUES ( compared to the Uniform-to-Random model ) • Expertise patterns INCREASE the ratio • Subcultural patterns DECREASE the ratio • Perspectival patterns DECREASE the ratio • w/r/to MEAN 1st FACTOR LOADING • Distributional patterns have little effect on this indicator, AND consensus analysis estimates actual competence very well … with one exception: • Expertise (triangular) pattern INFLATES mean competence as well as the ratio of eigenvalues … (because it violates the “homogeneity of items” assumption?)

27. Expertise (rectangular) pattern • The range of expertise about the same average competence also makes a difference: greater range  larger ratio of eigenvalues • Subcultural patterns • As expected, systematic differences in sub-group knowledge undermine consensus: • “By question” sub-groups may still show consensus overall, with the groups showing up on the 2nd factor • Different “answer keys” just destroy consensus • “Formal consensus model” (on dichotomized data) and “informal consensus model” yield very similar results

28. CONCLUSIONS

29. “Lessons” for Researchers • Since the ratio of eigenvalues is particularly sensitive to the distributional pattern of knowledge, REPORT MORE than just the ratio • Minimally, include: • Ratio of 1st to 2nd eigenvalues • Mean 1st factor loading (and st.dev. of those loadings) • Number of negative loadings • And, comparable “guidelines” should be established for evaluating these additional measures: e.g., 0.500 for mean loading; fewer than ~5% negative loadings in sample • These output statistics are necessary for more meaningful interpretations of one’s data

30. IF your data show a hefty mean 1st factor loading but a low ratio of eigenvalues… DO NOT leap to the conclusion that either (a) subcultures exist or (b) there is free variation in the domain • You may be dealing with a case of PERSPECTIVAL DIVERSITY … which would warrant further investigation, such as examining the inter-person correlation matrix and the response-profiles of individuals one at a time to see if you can detect a subtle social patterning to who-knows-what

31. Try to formulate questions that are “EQUALLY DIFFICULT” ( and ask lots of questions ) • Violations of Assumption 3 will inflate both the obtained ratio of eigenvalues & the mean 1st factor loading • e.g., Expertise (triangular) pattern INFLATES both indicators • So … ex post facto…if you notice that some questions were “much easier” than others, then either: (a) use higher threshold criteria before claiming the data conform to the cultural consensus model, and/or (b) remove the very easy questions and re-analyze

32. Future Directions • Developing additional ‘geometries’ of perspectival overlapping • Analyzing relations between a variety of measures describing the initial Resp x Resp correlation matrix and the key indicators from consensus analysis • Exploring different instantiations of “guessing” (binomial, truncated-normal, beta distributions) • Exploring other possible measures from the factor analysis as predictors of culture-sharing, e.g., 1st eigenvalue divided by sample size

33. Thank you … and we would be happy to continuetalking with interested folksafter the session

34. Scalar data analyzed via Informal Method (vertical axis)VSDichotomized data analyzed via Formal Method (horizontal axis) Mean 1st factor loadings( r = .969 ) Ratios of eigenvalues( r = .933 )

35. “Real” Survey Item “Real” Survey Item Simulated Item POTENTIAL PROBLEM… Frequency distributions of itemsfrom “real” surveys (top panels)are more graded than our“simulated” data (lower right) something we’re trying to resolve,but not there yet …