Presented by Daphne Minner Alina Martinez Abt Associates June 15, 2012

1. Community for Advancing Discovery Research in Education (CADRE)A Review of Research Instruments for STEM Teacher Practices, PCK, and Content Knowledge Presented by Daphne Minner Alina Martinez Abt Associates June 15, 2012 DR-K12 Principal Investigator Annual Meeting Arlington, VA

2. DR K-12 Portfolio Overview

3. Data from 248 Projects • Includes projects that submitted at least a proposal • Sources: • Provided by PIs voluntarily in response to requests • Found in the public domain

4. Project Materials were Reviewed and Coded

5. RFP Guides Innovation in STEM Education

6. Disciplines Addressed in Projects

7. Target Segment of the Education Population

8. Outcomes Studied

9. Types of Student and Teacher Outcomes Studied Students Teachers Achievement

10. Teacher Instrument Study Overview Rationale Methods

11. 75 projects assessed teacher outcomes with existing instruments Table A Table B

12. 82 unique instruments identified 42 used to assess Practices Reformed Teaching Observation Protocol (modified)—15 studies Inside the Classroom Observation and Analytic Protocol—8 Surveys of Enacted Curriculum (modified)—5 24 used to assess Content Views of Nature of Science (Form C)—3 studies National Assessment of Educational Progress (modified)—3 Praxis content tests/ Earth & physical science (modified)—2 TIMSS content tests (modified)—2

13. 82 instruments identified 22 used to assess Pedagogical Content Knowledge Mathematical Knowledge for Teaching—14 studies Knowledge of Algebra for Teaching (modified)—2 Science Teaching Efficacy Belief Instrument—2 Reformed Teaching Observation Protocol (modified)—2

14. More Refined Look at Constructs Instruction practices Content knowledge Instructional beliefs; teaching self-efficacy Class management Assessment practices/ skills Classroom social context Classroom physical context Administrative or policy context Student / teacher demographics Instructional planning

15. Instructional Practices

16. Instructional Practices plus

17. Teachers’ Content Knowledge

18. Teachers’ Content Knowledge

19. Instructional Beliefs

20. Multidimensional

21. System-wide Reform Focused

23. Dialogue Focused

24. Reliability Information • Reliability type • missing information (19, 38%) • internal stability alpha (11, 22%) (6 acceptable, 5 good) • interrater, % agreement (7, 14%) (2 low, 2 accept., 3 good) • interrater, Kappa (2, 4%) (acceptable) • Interrater, Spearman rank-order correlation (1, good) • Other (2, good and N/A) • More than one type (8, 16%) (all acceptable to good) • Level • missing information • <.59 or equivalent: low • .6-.79 or equivalent: acceptable • >.8 or equivalent: good

25. Validity Evidence • Validity Type • Missing information (28, 54%) • Content/ face (11, 21%) • Criterion/ predictive (9, 17%) • Construct (8, 15%) • Concurrent/ convergent (5, 10%) • Discriminant /divergent (2, 4%)

26. Trends • Instructional practices (n=22) • 11 instruments predominantly observation • 11 instruments with 1-2 other constructs, observation supplemented with survey methods • Teachers’ content knowledge (n=25) • primarily assessed domain-specific conceptual or procedural knowledge by multiple choice test • majority of these instruments being used for teachers were originally designed for students • 8 science, 3 math, 1 technology-specific tests

27. Trends cont. • Instructional beliefs (n=9) • Predominated by survey methods • 6/9 were science specific with only 2 for math • Multidimensional (n=23) • System-wide reform (predominantly surveys) (10) • Dialogue focused observation instruments to capture classroom climate and social aspects (13) • Making reliability and validity evidence more readily available with information about instruments is essential

28. Discussion Surprises? Gaps? Strategies? Resources?

29. Contact InformationAbt Associates