1 / 30

Developing General Education Course Assessment Measures

Developing General Education Course Assessment Measures. Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/. Why Validity & Reliability ?.

brooke
Download Presentation

Developing General Education Course Assessment Measures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing General Education Course Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/

  2. Why Validity & Reliability ? • Assessment results must represent student achievement of course learning objectives • Evaluation of the validity and reliability of the assessment measure provides the evidence that it does so

  3. Types of Measures • ‘Performance’ Measures • ‘Objective’ Measures

  4. Validity for Performance Measures • Identified learning outcomes represent the course (domain sampling) • The measure addresses the learning outcomes (content validity) • There is a match between the measure and the rubric (criteria for evaluating performance) • Rubric scores can be linked to the learning outcomes, and indicate the degree of student achievement within the course

  5. Validity for Objective Measures • Identified learning outcomes represent the course (domain sampling) • The items on the measure address specific learning outcomes (content validity) • Scores on the measure can be applied to the learning outcomes, and indicate the degree of student achievement within the course

  6. Content Validity (MA23)

  7. Content Validity (MA23)

  8. Content Validity (MA23)

  9. Content Validity (MA61)

  10. Content Validity (MA61)

  11. Objective I II III Description Identify the basic methods of data collection Demonstrate an understanding of basic sociological concepts and social processes that shape human behavior Apply sociological theories to current social issues Content Validity (SO11) A 30-item test measured students’ mastery of the objectives

  12. Content Validity (SO11)

  13. Content Validity (SO11)

  14. Content Validity (SO11)

  15. Reliability • Can it be done consistently? • Can the rubric be applied consistently across raters -- Inter-rater reliability • Can each of the items act consistently as a measure of the construct -- Inter-item reliability

  16. Inter-Rater Reliability (MA23) – The Rubric Item 1A

  17. Inter-Rater Reliability (MA23) – The Rubric Item 1B

  18. Inter-Rater Reliability (MA23) – The Data Set

  19. Inter-Rater Reliability (MA23) – Results

  20. Inter-Item Reliability (MA61)

  21. Face Validity and Reliability… Is this enough? • Measures with face validity & adequate levels of reliability can produce misleading/inaccurate results. • Even content valid measures cannot guarantee accurate estimates of student achievement MA23 – earlier pilot

  22. Criterion-Related Validity (MA23 – earlier pilot)

  23. Criterion-Related Validity (MA23)

  24. Criterion-Related Validity (MA61)

  25. Motivational Comparison (PC11) • 2 Groups Graded Embedded Questions Non-Graded Form & Motivational Speech • Mundane Realism

  26. Motivational Comparison (PC11) • Graded condition produces higher scores (t(78) = 5.62, p < .001). • Large effect size (d = 1.27).

  27. Motivational Comparison (PC11) • Minimum competency 70% or better • Graded condition produces greater competency (Z = 5.69, p < .001).

  28. Motivational Comparison (PC11) • In the non-graded condition this measure is neither reliable nor valid KR-20N-g = 0.29

  29. Motivational Comparison (PC11)

  30. Developing General Education Course Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/

More Related