Using Evaluation Methods to Monitor Implementation and Impact of Competency Based Pathways

1. Using Evaluation Methods to Monitor Implementation and Impact of Competency Based Pathways Jim Lindsay, Ph.D. Senior Researcher December, 2011

2. One More Thing to Plan: Evaluation • How well is your competency-based pathway model running? • Fidelity of implementation • What parts of model need additional attention? • Formative information • Are competency-based pathways having an impact on student learning? • Outcomes data + research design/staged roll-out help show impact.

3. Presentation on Evaluation Methods • Using data to assess fidelity and identify parts of pathway model that need improvement. • Thinking about impacts: Developing a Theory of Action. • Examining outcomes and impacts: Data and design.

4. Fidelity of Implementation • Determining the essential components of competency-based pathway. Examples: • Community and faculty buy-in • Teacher attendance at PD sessions • Specification of learning objectives • Assessments for competencies • Data system: immediate feedback to students • Teacher assignment • Students having difficulty receive more attention

5. Fidelity of Implementation • Examples at school/district level: • Administrator/team reviews students’ progress through objectives • Administrator/team meets with teachers to discuss student progression • Students getting awarded credits.

6. Assessing Fidelity • For each essential component, identify a data element that reflects whether component is working • For that data element, create a threshold that signifies “adequate” implementation. • Data element may be as simple as “presence” /”absence” of something

7. Assessing Fidelity • For each essential component, identify a data element that reflects whether component is working • For that data element, create a threshold that signifies “adequate” implementation. • Data element may just be as simple as “presence” /”absence” of something

8. Assessing Fidelity

9. Assessing Fidelity Fidelity = sum of scores/number 7/10 (70%)

10. Assessing Fidelity Components needing attention (70%)

11. Thinking Impact: Theory of Action • Figure that shows anticipated benefits of implementing pathways. • Immediate outcomes • Effects that occur while students participate in course/pathway or shortly thereafter • Longer-term outcomes • “ultimate” outcomes--tend to be ones of most interest to stakeholders • Figure helps planning team think about data that will indicate impact of pathways.

12. Thinking Impact: Theory of Action Program Immediate Outcomes Long-Term Outcomes Competency-Based Pathways - Specification of sequence of learning objectives for courses - Students direct pace of completion of learning objectives - Differential support to students based on their need Student motivation School Completion Student Attitudes Toward Subject Credit accumulation Graduation rate Time to complete course Number of students re-enrolling in CB pathways Broader Learning SAT/ACT- scores Student Engagement Other standardized achievement scores Learning Course Content Post-Secondary Enrollment Fidelity Information Scores on EOC Exams Enrollment in AP courses Utilization % taking SAT/ACT % of students applying to college

13. Examining Outcomes and Impact • Impact: Is initiative effective? •  Clear Comparison Group! • At school, classroom level, or student level • Preference: Random assignment to conditions, • Pre-post not good enough •  other things may contribute to pre-post differences. • Easy comparison group: • Schools/classrooms that implement 1-2 years later. • Schools that implement first “work out the kinks” • Impact: Is initiative effective? • Ability to make strong conclusions about whether initiative “works”

14. Example Theory of Action Program Immediate Outcomes Long-Term Outcomes Competency-Based Pathways - Specification of sequence of learning objectives for courses - Students direct pace of completion of learning objectives - Differential support to students based on their need Student motivation School Completion Student Attitudes Toward Subject Graduation rate Time to complete course Number of students re-enrolling in CB pathways Broader Learning SAT/ACT- scores Student Engagement Post-Secondary Enrollment Fidelity Information Learning Course Content Enrollment in AP courses Scores on EOC Exams Utilization % of students applying to college

15. Illustrating Impact **

16. District A Cannot answer “how is it working?” question. Unable to find particular components that aren’t working New superintendent- “is CBE effective”? Unable to answer District B Have data available to show stakeholders how it is working. Can determine which components need refinement. Can provide strong evidence of whether CBE is effective. Advantages of Considering Evaluation Now

17. Jim Lindsay, Ph.D. P:630-649-6591 >F:630-649-6700 E-Mail:jlindsay@air.org REL Midwest 1120 East Diehl Road, Suite 200 Naperville, IL 60563-1486 General Information:866-730-6735