Cultural Nuances, Assumptions and the Butterfly Effect: A Prelude

1. Cultural Nuances, Assumptions and the Butterfly Effect:A Prelude A Brief Introduction to Generalizability Theory for the Uninitiated

2. What is Generalizability Theory? Generalizability Theory—the theory that deals with the degree to which one can say that conclusions drawn on the basis of data collected from sampling different domains can be applied with confidence to those domains.

3. Still Lost? What we’re talking about you best know as instrument reliability. Commonly known examples: • inter-rater reliability, • test-retest reliability, • split-halves reliability, • Kuder-Richardson reliabilities, • Hoyt reliability, and • Cronbach’s alpha

4. Observed scores are composed of true scores and error. The variance of the observed scores is partitioned. Then estimates are combined to produce a coefficient. Xo = Xt + Xe X=score, t=true, o=observed, e=error So2 =St2 + Se2 S2 = variance r11 = St2/So2 Test (True) Score Theory

5. True Score Theory and Generalizability • Generalizabilty is an extension of true-score theory. • Sources of Variation are viewed from different perspectives. • S2O = S2LG + S2TG + S2LS + S2TS L=long term, T=temporary G=general, S=specific

6. So what does this have to do with Multivariate Analysis? All techniques we have studied deals with the degree of overlap of variance (information) for sets of variables (Tabachnick & Fidell, 2001). In the present case the overlap of interest is between observed variance for evaluators’ opinions about the spheres involved and the actual underlying dimensions influencing interpersonal interactions perceived by the evaluators--“true-score” variance. S2TOT = S2E + S2V + S2Sc + S2e E=evaluator, V=vignette,Sc=scale, e=error

7. Relationship of Gerneralizability to Multivariance To produce the variance estimates needed ANOVA is used. In this situation a 6 x 7 x 46 Evaluator (E) x Sphere + Primary Category (S) x Vignette (V) Mixed Effects Repeated Measures ANOVA is used. Evaluator and Vignette are considered random, Sphere fixed.

8. ANOVA DESIGNDoubly Repeated MeasuresScale and Vignette WithinEvaluator Between

9. Table 1 Variance Components Derived from the SPSS GLM repeated Measures Calculation for Overall Evaluations Source of VariationSS dfMS Vignette (V) 29.220 45 .649 Sphere (S) 639.865 6 106.644 Evaluator (E) 61.972 5 12.394 V x S 185.516 270 .687 V x E 96.528 225 .429 S x E 340.184 30 11.339 V x S x E Error (e) Residual 648.149 1350 .480

10. Table 2 • Variance Components and Expected Mean Squares for Overall Evaluations • Source of VariationMeans Square (MS)Expected Mean Square (EMS)Estimated Variance • Vignette (V) .649 2e + + 2VE + 2V .220 • Sphere (S) 106.644 2e + + 2VS + 2SE + 2S ≥94.618 • Evaluator (E) 12.394 2e + + 2VE + 2E 11.965 • V x S .687 2e + + 2VS ≤.207 • V x E .429 2e + + 2VE ≤-.059 • S x E 11.339 2e + + 2SE ≤10.859 • V x S x E 2e + 2VSE ≥.480 • Error (e) 2e ≥.480 • Residual .480 .480 • Total (TOT) 2e + 2S + 2V + 2E ≤107.763 • ≥107.283 • = S2S / S2TOT ≥ 94.618 / 107.763 = 0.879 0.879 ≤  ≤ 0.886 ≤ 95.094 / 107.283 = 0.886

11. Table 3 Variance Components Derived from the SPSS GLM repeated Measures Calculation for Spheres Source of VariationSS dfMS Vignette (V) 12.426 45 .276 Sphere (S) 22.575 5 4.515 Evaluator (E) 16.996 5 3.399 V x S 73.730 225 .328 V x E 27.643 225 .123 S x E 24.769 25 .991 V x S x E Error (e) Residual 168.092 1125 .149

12. Table 4 • Variance Components and Expected Mean Squares for Spheres • Source of VariationMeans Square (MS)Expected Mean Square (EMS)Estimated Variance • Vignette (V) .276 2e + + 2VE + 2V .153 • Sphere (S) 4.515 2e + + 2VS + 2SE + 2S ≥3.345 • Evaluator (E) 3.399 2e + + 2VE + 2E 3.276 • V x S .328 2e + + 2VS ≤.179 • V x E .123 2e + + 2VE ≤-.026 • S x E .991 2e + + 2SE ≤.842 • V x S x E 2e + 2VSE ≥.149 • Error (e) 2e ≥.149 • Residual .149 .149 • Total (TOT) 2e + 2S + 2V + 2E ≤6.923 • ≥7.072 • = S2S / S2TOT ≥ 3.345 / 7.072 = 0.473 0.473 ≤  ≤ 0.505 ≤ 3.494 / 6.923 = 0.505

13. A Sample Calculation As an interesting example of how the ’s are calculated here is one (see Table 2 for the data) Sphere (MS) = 106.644 2e + + 2VS + 2SE + 2S -V x S (MS) = .687 2e + + 2VS -S x E (MS) = 11.339 2e + + 2SE +Error (MS) ≤ .480 2e Sphere (Variance) ≤ 95.094 2S -Error (MS) ≤ .480 2e Sphere (Variance) ≥ 94.618 Total (TOT) ≤ 107.763 2e + 2S + 2V + 2E ≥ 107.283 = S2S / S2TOT ≥ 94.618 / 107.763 = 0.879 ≤ 95.094 / 107.283 = 0.886 0.879 ≤  ≤ 0.882

14. The Conclusion What we get is three little numbers that are not as simple as they look: • Primary Influence:  = .966 • Spheres:  = .473 • Overall:  = .879