150 likes | 266 Views
A Meta-Analysis Reliability Generalization Study: Reliability of the Statistical Anxiety Rating Scale Subscale Worth of Statistics. Gianna Rendina-Gobioff Meta-Analysis: Spring 2004. Research Domain. Reliability Incorrectly reported “The test is reliable”
E N D
A Meta-Analysis Reliability Generalization Study:Reliability of the Statistical Anxiety Rating Scale Subscale Worth of Statistics Gianna Rendina-Gobioff Meta-Analysis: Spring 2004
Research Domain • Reliability • Incorrectly reported • “The test is reliable” • “Reliability induction” – reporting reliability estimates from test publishers or literature instead of for the sample of interest in the study • Reporting problems • Reliability dependent on sample characteristics (more heterogeneous samples will have higher correlations/estimates) • Accuracy of effect sizes reported is in question (if reliability is poor, the effect size is low)
Research Domain • Statistical Anxiety Rating Scale (STARS) • Total of 51 likert 5-point response items • Statistics anxiety measure with six subscales: • Worth of statistics (16 items) • Interpretation anxiety (11 items) • Test and class anxiety (8 items) • Computational self-concept (7 items) • Fear of asking for help (4 items) • Fear of statistics teacher (5 items)
Research Questions: • What is a typical score reliability for the Worth of Statistics subscale? • Do the reliability estimates for the STARS Worth of Statistics subscale vary across sample disciplines of study?
Data Collection: • Search • ERIC, PsychINFO, and ISI Web of Science • Contacted researchers • Inclusion/exclusion criteria: • Timeframe: 1985 - present • Measure: STARS only • Required Information: coefficient alpha • Language: English only • Type of Publication: No restriction
Data Collection • Coding: • Dissemination: Journal or conference • Discipline of sample: Multi or education only • Education: Graduate or undergraduate/graduate mix • Reliability induction: Yes or no • Cronbach alpha: Subscales, total, range, median • Inter-rater reliability • Moderator of interest, discipline, reliability estimate of 0.91 (Cohen’s Kappa)
Data Analysis • Cronbach alpha transformed to z: • Weighted with inverse of Cronbach alpha variance: • Where i is the number of items and N is the sample size • Lipsey and Wilson (2001) fixed effects with moderator
Results • Study search results: • Started with 39 studies • Out of 39 obtained studies only 10 included • 17 – Duplications, not applicable, descriptive, or unavailable • 7 – Reliability induction • 3 – Did not report alpha for Worth subscale • 1 – Used revised version of instrument • 1 – Sample overlap • The 10 studies yielded 11 effect sizes
Evaluation of Analyses and Validity • Analysis concerns: • New weight methodology • Should have used mixed effects because random effects remained after including the moderator – Didn’t know how to run this analysis • Threats to Validity: • Apples/oranges minimized • Independence of effect sizes met • File drawer of concern • Methodological Rigor potential problem
Conclusions • Internal consistency is high across various samples for the Worth of Statistics Subscale (M=0.9124) • Multi-Discipline samples produced higher estimates of the subscale reliability, compared to Education only samples • Implications: • Guidance for choosing a measure of statistics anxiety - should take discipline of sample into consideration