1 / 22

Application of Generalizability Theory to Concept-Map Assessment Research

Application of Generalizability Theory to Concept-Map Assessment Research. Yue Yin & Richard J. Shavelson Stanford Educational Assessment Laboratory (SEAL) Stanford University & CRESST AERA 2004, San Diego CA. Overview.

aphrodite
Download Presentation

Application of Generalizability Theory to Concept-Map Assessment Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Application of Generalizability Theory to Concept-Map Assessment Research Yue Yin & Richard J. Shavelson Stanford Educational Assessment Laboratory (SEAL) Stanford University & CRESST AERA 2004, San Diego CA

  2. Overview • Part 1: Feasibility of applying G-theory to concept-map assessment (CMA) research - Examining the dependability of CMA scores - Designing a CMA for a particular application - Narrowing down alternatives • Part 2: Empirical study of using G-theory to compare two CMAs: - Construct-a-map with created linking phrases (C) - Construct-a-map with selected linking phrases (S)

  3. Concepts/Terms Linking lines Linking Phrases Proposition A Concept-map

  4. Variations in CMA

  5. Part 1 Feasibility of Applying G Theory to CMA Research

  6. Viewing CMA with G theory • Basic idea A particular type of score, given by a particular rater, based on a particular type of concept map, on a particular occasion, … is a sample from a multifaceted universe. • Object of measurement People—the variation in students’ knowledge structure • Facets Task (concept & proposition), response format, scoring system, rater, occasion, …

  7. Concept-term sampling Proposition sampling Rater sampling Occasion sampling Equivalence of alternate forms Internal consistency Inter-rater reliability Stability over time G theory vs. CTT Similarity G Theory’s Advantage • Integrate conceptually and simultaneously evaluate all the technical properties above • Estimate not only the effect of individual facets, but also interaction effects • Permits us to optimize an assessment’s technical quality

  8. Examining Technical Properties & Designing Assessments • Examining dependability (G study) How well can a measure of student’s declarative knowledge structure be generalized across concept map tasks? scoring systems? occasions? raters? propositions? different concept samples? • Designing an assessment (D study) How many concept map tasks, scoring systems, occasions, raters, propositions, and/or different concept samples will be needed to obtain a reliable measurement of students’ declarative knowledge structure?

  9. Narrowing Down Alternatives • Task - Which task type is more reliable over raters, occasions, propositions, concept samples? - Accordingly, this task needs fewer raters, occasions, propositions, and concept samples. • Scoring system - Which scoring system is more reliable over raters, occasions, propositions, concept samples? - Accordingly, this scoring system needs fewer raters, occasions, propositions, and concept samples.

  10. Part 2 Empirical Study of Using G-theory to Compare Two CMAs

  11. Two Frequently UsedCMAs • Construct-a-map with created linking phrases (C)--Provides a cognitively valid measure of knowledge structure (e.g., Ruiz-Primo et al., 2001 & Yin et al., 2004) • Construct-a-map with selected linking phrases (S)--Provides an efficient way to measure knowledge structure (e.g., Klein et al., 2001)

  12. Method • Concept-map task - 9 Concepts (for C & S) water, volume, cubic centimeter, wood, density, mass, buoyancy, gram, and matter - 6 Linking phrases (for S only) is a measure of… has a property of… depends on… is a form of… is mass divided by… divided by volume equals… • Participants - 92 eighth-graders - 46 girls - previously studied a related unit - no related instruction between two occasions • Procedures C  S (n = 22) S  C (n = 23) C  C (n = 26) S  S (n = 21)

  13. Water Wood has has is a form of is a form of has has has has Matter has has a property of has a property of has a property of Mass divided by volume equals Density is mass divided by Volume is unit of is a unit of depends on Gram CC Buoyancy Criterion Map

  14. Mandatory Propositions

  15. CS & SC Person (P) Proposition/Item (I) Format (F) P x F P x I F x I P x F x I, e CC & SS Person (P) Proposition/Item (I) Occasion (O) P x O P x I O x I P x O x I, e Source of Variation

  16. Variance Component Estimate

  17. G Study in SC & CS

  18. G Study in CC & SS

  19. D Study for C CMA

  20. D Study for S CMA

  21. Conclusions • G study pinpoints multiple sources of measurement error, thereby giving insight into how to improve the reliability and applicability of CMA via a D study • C and S mapping tasks are not equivalent in their technical properties • Fewer occasions and propositions are needed in S than C to get a reliable evaluation of students’ declarative knowledge structure

  22. Thank You for Your Interest!  To get the complete paper, please either contact Yue Yin at yyin@stanford.edu Or download the file directly at http://www.stanford.edu/dept/SUSE/SEAL/

More Related