1 / 24

Developing Valid Rubrics for Assessing Global Awareness and Global Perspective

Developing Valid Rubrics for Assessing Global Awareness and Global Perspective. Dr. Stephanie Doscher Florida International University. AAC&U General Education and Assessment Conference Boston, Massachusetts, March 1 st , 2013. Session Outline. Global Learning @ FIU Rubric anatomy

samson
Download Presentation

Developing Valid Rubrics for Assessing Global Awareness and Global Perspective

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education and Assessment Conference Boston, Massachusetts, March 1st, 2013

  2. Session Outline • Global Learning @ FIU • Rubric anatomy • Developing the rubrics • Scorer training • “Rubric for rubrics” • Gathering evidence of validity and reliability

  3. Text

  4. FIU’s Global Learning SLOs Global Awareness: knowledge of the interrelatedness of local, global, international and intercultural issues, trends, and systems Global Perspective: ability to develop a multi-perspective analysis of local, global, international, and intercultural problems Global Engagement: willingness to engage in local, global, international, and intercultural problem solving

  5. Active Learning Strategies/ Performance Assessments • Essay, story, or poem • Research paper • Literary analysis • Book or article review • Case study • Speech • Journal response • Art exhibit • Portfolio • Musical composition • Lab report • Strategic planning • In-class discussion • Editorials • Peer editing • Poster presentation • Video • Podcast • Mock trial • Oral presentation • Debate • Role play • Online discussion • Blogs • Advertising campaign • Building a prototype • Modeling • Experiments • Service learning

  6. Developing the Rubrics • Faculty learning community • Pilot case studies, questions, scoring criteria • Revisions based on response trends • Pilot study • Benchmark responses • Anchor papers • Faculty feedback, expert judge feedback • Two field tests • Scorer training • Rubric language, response minimum

  7. Scorer Training • Full-time and adjunct faculty • Pre-training packet • Open discussion and review of cases, questions, and rubrics • Norming session with anchor papers • Sample scoring session, 10% of total papers

  8. Validating the Rubrics Validity: Can the rubrics detect the differences in students’ development of global awareness and global perspective? Reliability: Can trained faculty raters agree on rubric scores 80% of the time or more?

  9. Validating the Rubrics Research Design Pretest Treatment Posttest Global learning O1 X1 O1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Non-global learning O2 X2 O2

  10. Data Collection • Emails to chairs and potential faculty • Pretest—within first two weeks of class • Posttest—within last two weeks of class • Trained faculty raters • Two raters scored each question, third rater for discrepant scores

  11. Results Inter-rater reliability • Global awareness rubric • Pretest (.89, p < .0001) • Posttest (.95, p < .0001) • Global perspective rubric • Pretest (.92, p < .0001) • Posttest (.91, p < .0001)

  12. Results Validity – Global Awareness Rubric • No significant main effects • Post-hoc analysis: Groups differed significantly on pretest scores, p=.003 • Global learning (M = 1.51) • Non-global learning (M = 1.85) • Significant interaction between global awareness pretest score and the treatment in predicting global awareness posttest score, p = .005

  13. Global Awareness Rubric

  14. Results Validity – Global Perspective Rubric • No significant main effects • Post-hoc analysis: Groups differed significantly on pretest scores, p =.003 • Global learning (M = .90) • Non-global learning (M = 1.2) • Significant interaction between global perspective pretest score and the treatment in predicting global perspective posttest score, p = .005

  15. Results

  16. Interpretation of Results Reliability • Both rubrics highly reliable • Consistent with literature on rubric development and rater training • Empirical support for structural validity

  17. Interpretation of Results Validity • Rubric detects differences between global learning and non-global learning students • Rubric detects differences within the group of global learning students

  18. Thank You! For more information, please contact me: Stephanie Doscher, Office of Global Learning sdoscher@fiu.edu Visit our web site: goglobal.fiu.edu

More Related