21 st Century Skills: What Can Be Measured?

1. 21st Century Skills: What Can Be Measured? Beverly Farr, Director of Evaluation MPR Associates

2. A Tale of Two Projects: Evaluation of 1) ConnectEd Demonstration Sites 2) National Ford PAS Network

3. Evaluation of the ConnectEd Network of Demonstration Sites (n=16) • Provide evidence of effectiveness on a set of student outcome indicators • Showcase well-designed examples of multiple pathways or pathway components

4. Multiple pathways • Comprehensive, coherent strategy designed to re-engage high school students in serious learning • Prepare students for postsecondary education and career  • Connect academics to real-world applications

5. Multiple Pathways • Lead to the full range of postsecondary opportunities. • Improve student achievement. Designed to produce higher academic and technical achievement and higher rates of high school completion and to support the development of students’ critical-thinking and problem-solving, communication, and collaboration skills.

6. Evaluation Challenge Gary Hoachlander (2005) points out, “Standardized tests . . . do not measure students’ diagnostic abilities, capacities for bringing interdisciplinary knowledge to bear on complex problems, understanding of systems, or facility in applying abstract knowledge and academic skills to authentic, real-life situations” (p. 38).

7. Evaluation Components • Collection of (2007-08) student data • Participant data (number, grade level, demographics) • CAHSEE and CST scores/proficiency levels • Attendance, continuation, PSE readiness and plans • Site visits during fall/winter of 2008 • Interviews (school and district staff, students, partners) • Classroom observations • Review of documents

8. Student Outcomes: Results on Indicator Data • DMD • Palmdale • HPHS

9. Student Outcomes • Attendance, promotion, and continuation rates • 10th-grade CAHSEE pass rates • California Standards Tests (CSTs) • Student attitudes, behaviors, and career awareness • PSE readiness and intentions

10. Graduation and a-g course fulfillment

11. Science Achievement

12. School of Digital Media & Design The School of Digital Media and Design focuses on creating an environment where students interested in multimedia, visual arts and production will master high-level communication, technological and critical thinking skills. Classes emphasize multimedia and collaborative projects to prepare students for lifelong learning, a fulfilling career, and a successful future.

13. Math Achievement

14. English Achievement

15. Other Reported Outcomes • DMD • Palmdale • HPHS

16. Qualitative Findings • Student Attitudes • Behaviors and Skills • Awareness of Career Options • Workplace Readiness • College Readiness

17. Purpose of the Evaluation of Ford PAS: Conduct rigorous study on the effects of the program on student academic achievement, attitudes, and 21st century skills

18. Research Questions • What impact does the program have on students in terms of: a) cognitive performance and academic/business knowledge and skills; b) affective student factors such as attitudes, behavior, future career and vocational aspirations; c) future educational pursuits and success in those settings?

19. Research Questions • In what ways has the program helped students connect classroom learning with real-world experiences? To what extent are students able to apply their academic/business knowledge and skills to real-world settings?

20. Results of Implementation Survey

21. Reports by Teachers and Site Coordinators on Student Outcomes

22. Planning for Measurement of Student Outcomes in Ford PAS Study

23. Planning Analyses • Elaboration of Constructs • Cross-walk of Ford PAS modules and core skills • Review of Identification of Essential Knowledge for 21st Century by Partnership • Matrix of Available Assessments in Use or in Development

24. Challenges and Next Steps