Using Concept Maps for Evaluation

1. Using Concept Maps for Evaluation Combines results from 2 projects conducted across 2 years: Using Concept Maps Transcribed from Interviews to Quantify the Structure of Preschool Children’s Knowledge About Plants (Wehry, S., Algina, J., Hunter, J., & Monroe-Ossi, H.) & Healthy Habits Through Literacy: A Concept Mapping Curriculum for Preschool and Prekindergarten Children (Monroe-Ossi, H.,Wehry, S., Algina, J., & Hunter, J.) The Florida Institute of Education at the University of North Florida

2. Using Concept Maps for Evaluation • A shared goal of the Young Florida Naturalists & Healthy Habits Through Literacy projects – examine concept mapping as a tool to quantify conceptual development in preschool children. • Researchers need a reliable and valid way to use children’s concept maps for program evaluation.

3. Scoring System: 3 Components Propositions • 0 points if irrelevant • 1 point if an example of concept • 2 points if describes an attribute of concept • 3 points if states a purpose of the attribute Cross-Links • 5 points each Hierarchical Levels • 5 points each

4. Scoring System: Examples

5. Interrater Reliability Study • Data: Young Florida Naturalists Assessment 2 Interviews (n = 48). • Maps: 1 researcher mapped all interviews. • Raters: 3 trained researchers rated all maps. • Analysis: Estimated how much variance was accounted for by raters, maps, and raters*maps.

6. Interrater Reliability Study: Results

7. Concept Mapping Protocol • The 3 Young Florida Naturalist raters used the 48 interviews to develop a mapping protocol. • Propositions do not have to be grammatically correct. • Do not map irrelevant propositions. • Mapped propositions can stand alone, that is, not connected to the focal concept. • When in doubt, the benefit goes to the child. • Score the resulting concept maps to test ease of scoring.

8. Interrater & Intermapper Reliability Study • Data: Healthy Habits Through Literacy Assessment 2 Interviews (n = 48). • Maps: 3 researchers mapped all interviews. • Raters: 3 trained researchers rated all maps. • Analysis: Estimated how much variance was accounted for by the interviews, mappers, raters, and all interactions of the 3 sources.

9. Interrater & Intermapper Reliability: Results

10. Validity • Young Florida Naturalist - Prekindergarten children were assessed in the fall and spring (n = 26). • The Test of Early Reading Ability -Third Edition (TERA-3) • Bracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R) • Average Concept Map Scores

11. Validity Study: ResultsCorrelations with Concept Map Scores Note: *p < .10; ** p < .05; SRC is the School Readiness Composite.

12. Validity: Regression Results • After controlling for gender, age, prekindergarten class, and pretest score, the children’s concept map scores predicted spring: • BBSC - 3:R Texture/Material scale achievement. • BBSC - 3:R Self-/Social Awareness scale achievement.

13. Validity 2: Measures • Healthy Habits Through Literature -Prekindergarten and preschool children were assessed in the spring (n = 35) using the following measures: • The Peabody Picture Vocabulary Test - Fourth Edition (PPVT-4). • The Expressive Vocabulary Test – Second Edition (EVT-2). • Bracken Basic Concept Scale -Third Edition: Receptive (BBCS-3:R). • Bracken Basic Concept Scale -Third Edition: Expressive (BBCS-3:E). • Average Concept Map Scores.

14. Validity 2: ResultsCorrelations with Concept Map Scores Note: *p < .10; ** p < .05; SRC is the School Readiness Composite

15. Conclusions • Concept maps developed from preschool children’s transcribed interviews can be reliably scored. • Preschool children’s transcribed interviews can be reliably mapped and scored. • Resulting concept map scores correlated with measures of expressive vocabulary, attributes of people & objects, and alphabet knowledge.

16. Discussion Revise Scoring System • 1. To reflect the use of Dynamicvs. Static Statements. • 2. Concerning the use of examples. Revise Mapping Protocol • 1. To reduce variance due to mapper in cross- link & hierarchical level scores. • 2. To better fit across curricula.

17. Discussion • Why rate concept maps rather the interviews? • Rating the interview did not provide clarity in determining the structure of children’s knowledge relative to the domain (Novak & Musondo, 1991) • Concept mapping children’s interviews makes visible the their propositions, cross-links, and hierarchical structure. • The concept maps make visible places where instruction can be improved.

18. Next Steps • Professional Development: • 22 prekindergarten teachers to use concept maps to facilitate children’s learning. • Kindergarten teachers to use concept maps to help children connect their background knowledge to newly acquired information. • Middle School Implementation: • Use concept mapping strategies with 6th and 7th graders in after-school programs at two middle schools.