1 / 41

Two Methodological Issues concerning Consensus Analysis

This article discusses two methodological issues in consensus analysis: counterbalancing items and identifying the meaning of the 2nd factor. It provides examples and explores the impact of counterbalancing on consensus findings. Additionally, it examines how to determine the meaning of the 2nd factor through preliminary reminders.

tgranger
Download Presentation

Two Methodological Issues concerning Consensus Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Two Methodological Issues concerning Consensus Analysis John B. Gatewood Lehigh University SASci / SCCR meetings Feb 18-21, 2009, Las Vegas, Nevada

  2. The two issues • COUNTER-BALANCING ITEMS for consensus analysis • Examples: • Perceptions of Christmas program in Bethlehem, PA( Cameron & Gatewood 1990 ) • Employee perceptions of credit unions( Gatewood & Lowe 2004-2008 ) • Identifying the MEANING OF 2ND FACTOR from consensus analysis • Example: • Residents’ understandings of tourism in Turks and Caicos Islands( Gatewood & Cameron 2006-2009 )

  3. Counter-balancing items for consensus analysis …

  4. Entertaining Historic Boring Interesting Commercialized Meaningful Glitzy Nostalgic Serene Insufficient Old fashioned Ethnic Religious Tasteful High pressured Crowded Musical Small townish Enriching Authentic Example #1: The Christmas Program in Bethlehem, PA (ratings for 20 adjectives)

  5. Entertaining Historic Boring Interesting Commercialized Meaningful Glitzy Nostalgic Serene Insufficient Old fashioned Ethnic Religious Tasteful High pressured Crowded Musical Small townish Enriching Authentic Example #1: The Christmas Program in Bethlehem, PA (ratings for 20 adjectives)

  6. Original data …(item means and standard deviations)

  7. Same data with “negative” items inverted

  8. Evenly counter-balanced data(4 positive items inverted)

  9. Different consensus findings !

  10. Example #2: Cultural Model of Credit Unions (pilot vs. follow-up studies) • PILOT STUDY (2006) • Questionnaire developed BEFORE the cultural model • Ex post facto, only 14 items directly relevant to the cultural model’s elements … all phrased “positively” (using informants’ own words) • FOLLOW-UP STUDY (2008) • Expanded questionnaire AFTER constructing the cultural model • 50 counter-balanced items relevant to the cultural model’s elements … paired questions for each idea

  11. Strongly Agree Neutral Strongly Disagree Pilot Study …

  12. Follow-up Study …

  13. VERY different consensus findings !!

  14. WHY counter-balancing makes a difference Responses as z-scores: z = ( X – m ) / s • m = respondent’s mean across items • s = respondent’s st. dev. across items Pearson correlation coefficient: r = Σ (zx zy) / N COUNTER-BALANCING ITEMS … • increases within-respondent variances … more “undulations” in each person’s response-profile • Hence, counter-balancing makes higher correlations among respondents mathematically possible (although not necessary).[ Note: “correlation” is undefined between constants. ] • induces respondents to use more of the response-scale→ finer gradations of responses, more “interval-like” data

  15. CONCLUDING REMARKS #1 • Researchers using consensus analysis on rating data must be sure to “counter-balance” items • Goal = batteries of questions containing approximately equal numbers of “positive” and “negative” items • How? • INVERTING ITEMS ex post facto … a data transformation step • formulating PAIRED QUESTIONS in advance … an improvement to data collection

  16. Identifying the meaning of the 2nd consensus factor …

  17. Preliminary reminders … • The “informal method” of consensus analysis: • rating or ranking data (not categorical responses) • Pearson r is measure of “similarity” among response profiles … no adjustments for guessing • minimum residual factor analysis of the R x R correlation matrix (hence, factors may be correlated with one another) • 1st FACTOR = overall similarity among the response profiles (loadings indicate how well each person represents the entire sample) • 2nd FACTOR = the second largest source of pattern variability in the response profiles … “substantive meaning” is unknown in advance

  18. Example #3: Understandings of Tourism …(ratings on 119 counter-balanced items) • SPECIAL SAMPLE (N = 29) • Individuals we interviewed during first year • Questionnaire “grown” from these interviews, with 119 counter-balanced items (1-to-5 scale) • Same people also completed questionnaire during second year • RANDOM SAMPLE (N = 277) • Stratified random sample of registered voters who completed questionnaire … no additional information about them

  19. Cultural consensus within Random Sample • Weak cultural consensus with respect to the 119 similarly-formatted “cultural model items” • Random Sample (N=277) • Ratio of 1st to 2nd eigenvalues = 4.515 • Mean 1st factor loading = .499 • 9 negative loadings, or 3.2% of sample

  20. Disaggregating Random Sample improves consensus … mostly

  21. Cultural consensus within Special Sample • Weak cultural consensus in this group (N=29), too • Ratio of 1st to 2nd eigenvalues = 3.355 • Mean 1st factor loading = .584, with 0 negative loadings • Hence, use Special Sample to investigate the second largest source of variability… (2nd factor accounts for 21.6% of variance in this R x R correlation matrix) • Examining the 2nd factor loadings for these 29 familiar informants, we began to see a very interpretable pattern…

  22. Cluster 1(n=12) Cluster 2(n=17)

  23. JOHNSON’S HIERARCHICAL CLUSTERING (average method) Cluster 1 Cluster 2 A A A A A 1 A A A A A A 1 A | A A A A A A A A A A A A A A A A A 2 0 1 7 0 0 1 1 0 2 7 2 | 3 2 2 2 0 1 0 1 0 0 0 2 1 2 2 2 3 6 3 5 a 6 2 1 2 9 1 b 7 | 0 9 3 5 1 9 4 4 8 5 7 4 0 0 8 2 1 ------ - - - - - - - - - - - - | - - - - - - - - - - - - - - - - - 0.7129 . . . . . . . XXX . . . | . . . . . . . . . . . . . . . . . 0.6934 . . . . . . . XXX . . . | . . . . . . . . . . . . . . . XXX 0.6613 . . . . . . . XXX . . . | . . . . . . . . . . . . . . XXXXX 0.6417 . . . . . . XXXXX . . . | . . . . . . . . . . . . . . XXXXX 0.6060 . . . . . . XXXXX . . . | . . . . . . . . . XXX . . . XXXXX 0.6025 . . . . . XXXXXXX . . . | . . . . . . . . . XXX . . . XXXXX 0.5926 . . . . . XXXXXXX . . . | . . . . . . . . . XXX . . XXXXXXX 0.5754 . . . . . XXXXXXX . . . | . . . . XXX . . . XXX . . XXXXXXX 0.5694 . . . . . XXXXXXX . . . | . . . . XXX . . . XXXXX . XXXXXXX 0.5656 . . XXX . XXXXXXX . . . | . . . . XXX . . . XXXXX . XXXXXXX 0.5420 . . XXX . XXXXXXX . . . | . . . . XXX XXX . XXXXX . XXXXXXX 0.5290 . . XXX . XXXXXXX . . . | . . . . XXX XXX . XXXXX XXXXXXXXX 0.5282 . . XXX . XXXXXXX . . . | . . . . XXX XXX XXXXXXX XXXXXXXXX 0.5191 . . XXX . XXXXXXXXX . . | . . . . XXX XXX XXXXXXX XXXXXXXXX 0.5085 . . XXX . XXXXXXXXX . . | . . . . XXX XXX XXXXXXXXXXXXXXXXX 0.4899 . . XXX . XXXXXXXXX . . | . . . . XXX XXXXXXXXXXXXXXXXXXXXX 0.4688 . . XXX XXXXXXXXXXX . . | . . . . XXX XXXXXXXXXXXXXXXXXXXXX 0.4458 . . XXX XXXXXXXXXXX XXX | . . . . XXX XXXXXXXXXXXXXXXXXXXXX 0.4440 . . XXX XXXXXXXXXXX XXX | . . XXX XXX XXXXXXXXXXXXXXXXXXXXX 0.4327 . . XXX XXXXXXXXXXX XXX | . . XXX XXXXXXXXXXXXXXXXXXXXXXXXX 0.4132 . XXXXX XXXXXXXXXXX XXX | . . XXX XXXXXXXXXXXXXXXXXXXXXXXXX 0.3634 . XXXXX XXXXXXXXXXX XXX | . . XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.3483 . XXXXXXXXXXXXXXXXX XXX | . . XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.3380 . XXXXXXXXXXXXXXXXXXXXX | . . XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.3184 . XXXXXXXXXXXXXXXXXXXXX | . XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.3038 . XXXXXXXXXXXXXXXXXXXXX | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.2818 . XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.2241 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  24. “Subcultures” within Special Sample • Analyzing the clusters separately, consensus indicators go up sharply • Cluster 1 (n=12) • Ratio of 1st to 2nd eigenvalues = 7.061 • Mean 1st factor loading = .640, with no negative loadings • Cluster 2 (n=17) • Ratio of 1st to 2nd eigenvalues = 9.838 • Mean 1st factor loading = .653, with no negative loadings • Conclusion: there are two coherent viewpoints (different ‘answer keys’) in the Special Sample

  25. The two viewpoints (in Special Sample) • Based on the individuals who best represent each subcultural group (and taking into account the views expressed by them in interviews), the two viewpoints might be characterized as follows • Cluster 1: “Cautiously ambivalent” • Some concern about the long-term consequences of tourism; tourism involves a trade-off between good and bad impacts • Cluster 2: “Pro-tourism, pro-growth” • Very positive about changes tourism has wrought;pro-growth and pro-development; change is progress

  26. Survey items that differentiate the two viewpoints (in Special Sample) • Independent-samples t-tests on the 119 cultural model items in questionnaire (Cluster 1 vs. Cluster 2)… • 47 items show “statistically significant” group-group differences at the unadjusted α =.05 level • Conversely, the two groups did not differ significantly on 72 items… (reason the Special Sample, as a whole, shows weak consensus)

  27. The “usual suspects” don’t explain the viewpoints • NO difference with respect to: • Age; Sex; Education; Household income • How often think about tourism; Speak with tourists • Perceived overall financial benefit from tourism(Variable = self + family + neighbors + island + country) • Sources of information • Almost significant contrast (α =.057) : • Cluster 1 has traveled to more parts of the world • One significant contrast (α =.033) : • Cluster 2 reports more personal financial benefit from tourism(Variable = self + family)

  28. Extrapolating from Special Sample • EMPIRICAL QUESTION:Is there a similar “viewpoint” variation – the same sort of “subcultural” attitudinal variation – in the larger, Random Sample? • PRELIMINARY OBSERVATION:Overall, response profiles across the whole battery of 119 items are very similar between the Special Sample (as a whole) and the Random Sample … r = .938 • Note: Special Sample has greater variance among items means, but very similar pattern of up’s-and-down’s

  29. Both samples’ response profiles are very similar overall … (r = .938)

  30. Extrapolating…? – two approaches 1. Profile Matching • Compare each Random Sample respondent with the two “subcultural” response profiles (across 47 items) from the Special Sample • See whether these profile matching measures correlate with the Random Sample 2nd factor loadings from consensus analysis 2. Thematic Indices • Construct multi-item, additive indices to measure different themes that seem to distinguish the Special Sample’s two “viewpoints” • See whether one or more of these indices correlate with the 2nd factor loadings from consensus analysis (both samples)

  31. Profile matching approach Scatterplot: Random Sample’s correlations with respect to the Special Sample’s two “subcultural” response profiles

  32. “r2 – r1” … a computed variable from information depicted in the scatterplot, wherer1: Pearson r vis-à-vis Cluster 1’s response profiler2: Pearson r vis-à-vis Cluster 2’s response profile • Thus, • Positive values  respondent is more similar to the “Pro-Tourism” (Cluster 2) viewpoint • Negative values  respondent is more similar to the “Cautiously Ambivalent” (Cluster 1) viewpoint

  33. The attitudinal gradient found in the Special Sample also exists in the Random Sample • Correlation between the “r2 – r1” pattern-matching variable and the 2nd consensus factor scores for the Random Sample is VERY high … r = .903 • Thus, the Random Sample’s “second largest source of variation” is very similar to the attitudinal gradient identified in the Special Sample

  34. Thematic indices approach • Candidate items were selected from all 119 cultural model questions based on their face validity … subsequently, winnowed by standard criteria of index construction using Random Sample’s data • RESULT: Six additive indices … scaled to rangefrom 1-to-5 (1=maximally negative, 3=neutral, 5=maximally positive) • Social Impacts(7 items, Cronbach’s α = .780) • Heritage Optimism(5 items, Cronbach’s α = .737) • General Pro-Tourism Outlook(7 items, Cronbach’s α = .717) • Financial Impacts(5 items, Cronbach’s α = .704) • Environmental Impacts(5 items, Cronbach’s α = .673) • Orientation to Tourism Work(4 items, Cronbach’s α = .636)

  35. To our surprise (and delight), the six thematic indices could be combined to form a single, second-order index • MacroIndex … a two-stage additive index based on 33 items, Cronbach’s α = .812 • Histogram of MacroIndex scores for Random Sample (mean = 3.23)

  36. MacroIndex correlates VERY highly with consensus 2nd factor loadings (both samples) • MacroIndex scores are extremely highly correlated with the 2nd factor loadings from consensus analysis… • Random Sample (N=277) r = .922 • Special Sample (N=29) r = .975 • INTERPRETATION: • MacroIndex’s 33 constituent items virtually are the substantive issues that underlie the “second largest source of variation” among respondents • The attitudinal gradient first discovered in the Special Sample is also present (and now substantively identified) in the Random Sample

  37. [ Methodological aside … ] It was only by having a “Special Sample” – people we interviewed AND surveyed – that we: • became aware different viewpoints existed, • were prompted to investigate how these viewpoints are associated with distinguishable response patterns in the survey data

  38. CONCLUDING REMARKS #2 • When overall consensus indicators are low or mixed, the 2nd factor deserves detailed investigation to identify the “subcultural” variations • And, the substantive meaning of the 2nd consensus factor must be established anew for each dataset • Sometimes the 2nd factor may be correlated with demographic-biographic variables, e.g., sex, age, education, income, residence, political party, etc. • Sometimes not, or only weakly so • When demographic-biographic variables aren’t working, there are at least two other ways to proceed: • Profile matching (requires a “special sample”) • Constructing thematic indices

  39. Finis … ( now let’s talk about this stuff ! )

More Related