Evaluating the EHR Portfolio

1. Evaluating the EHR Portfolio • Judith A. Ramaley • Assistant Director • Education and Human Resources

2. Current Forms of Review • Committees of Visitors (focus on a single program) • GPRA Reports (focus primarily on individual awards; i.e., “nuggets”) • Third Party Evaluation (usually a single program) • None of these approaches encompasses a portfolio model

3. Concept of a Portfolio • “The securities held by an investor” • “A flexible case for carrying loose papers, pictures or pamphlets” • Source: Webster’s Ninth New Collegiate Dictionary

4. A Policy Framework • Defining the K-16 STEM Program Portfolio and its Relevance to Meeting National Needs • Addresses areas of critical national need • Furthers the NSF mission • Well designed • Promotes collaboration between education and research communities • Demonstrates accountability: produces research findings that translate theory into practice • Fosters implementation of successful models in new or expanded settings • Increases participation of underrepresented groups in STEM education and careers

5. Application of the Policy Framework • Portfolio Review • Review of Newly Proposed Programs • Program Accountability and Revitalizing the Portfolio

6. The NSF Mission • The Mission of the NSF is • “To promote the progress of science; to advance the national health, prosperity, and welfare;…and to initiate and support basic scientific research and programs to strengthen scientific research potential and science education programs at all levels (Source: National Science Foundation Act of 1950) • Through discovery, learning and innovation that will ensure that the nation has the people, ideas and tools necessary for a healthy science and engineering enterprise;

7. The NSF Mission • The Mission of the NSF is • Achieved through the core NSF-wide strategies of development of intellectual capital, integration of research and education, and the promotion of partnerships; • In order to (a) make it possible for the U.S. to uphold a position of world leadership in all aspects of science and engineering; (b) promote the discovery, integration, dissemination, and employment of new knowledge in service to society; and (c) achieve excellence in U.S. science, technology, engineering and mathematics education. (Source: Core Strategies to Realize the Outcome Goals, NSF GPRA Strategic Plan FY2001 – 2006.)

8. The EHR Portfolio EHR’s mission is to achieve excellence in U.S. science, technology, engineering and mathe- matics (STEM) education at all levels and in all settings (both formal and informal) in order to support the development of a diverse and well- prepared workforce of scientists, technicians, engineers, mathematicians and educators and a well-informed citizenry that have access to the ideas and tools of science and engineering. The purpose of these activities is to enhance the quality of life of all citizens and the health, prosperity, welfare and security of the nation.

9. Goals of the EHR Portfolio • Prepare the next generation of STEM professionals and attract more Americans to STEM careers. • Increase the technological and scientific literacy of all Americans so that they can exercise responsible citizenship in an increasingly technological society and acquire knowledge of science, mathematics and technology that is appropriate to the development of workforce skills and life-long career opportunities.

10. Goals of the EHR Portfolio • Broaden participation (diversity) and achievement in STEM. • Attend to critical workforce needs requiring significant math and science skills and knowledge, both by attracting new people to these STEM careers and by support for the development and retooling of the current STEM workforce.

11. Capacity-Building Strategies • Identify effective ways to prepare and support teachers and faculty who can inspire and challenge students in the STEM disciplines and to provide them with effective materials and strategies to promote and assess learning. • Invest in research in the science of learning, facilitating the translation of research into practice, and create supportive learning environments and STEM pathways by developing models of reform/systemic change at both institutional and multi-institutional levels through networking, partnerships, alliances and collaborations.

12. Capacity-Building Strategies • Ensure that the STEM community is broadly representative of the nation’s individuals, geographic regions, types of institutions and STEM disciplines. • Identify effective ways (formal and informal) to address the STEM knowledge requirements of adults so that they can be productive members of the workforce and informed and active citizens.

13. Elements of the EHR Portfolio • Supporting individuals to enhance the capabilities of individuals preparing for STEM careers (including K-12 teachers) and people currently in the workforce, and to attract a diverse group of people to become scientists, engineers, technologists, mathematicians and educators; • Developing ideas, tools and communication strategies that will enhance the quality of STEM education; inform all citizens about science, technology and mathematics; and enhance public understanding of current research and its implications; • Utilize a solid body of evidence created by an analysis of the existing research and examination of practice to guide further development of the NSF portfolio as well as to support the diffusion and implementation of strategies that promote educational excellence.

14. Elements of the EHR Portfolio • Supporting the development of links among networks of researchers and practitioners that help create the capacity to introduce and sustain change in order to enhance the quality of STEM education and broaden participation in STEM careers; and • Developing institutional environments and enhancing system level interactions and partnerships that support change and the introduction of strategies that will enhance the preparation and development of the STEM workforce – creating the infrastructure to support collaboration across disciplines and across institutions that promote excellence in STEM education and facilitate the application of innovation to economic development.

15. Evaluating the EHR Portfolio • Adopt a thematic approach • Utilize an external panel of researchers and educators • Conduct two reviews a year, each requiring approximately 6 months • Select topics based on changes in NSF priorities, emerging opportunities, strong public interest, and budget availability • First topic will be mathematics education (K-12 and transition to higher education) • Second topic will be use of technology in education • Additional topics to be developed in consultation with the EHR Advisory Committee

16. Purpose of the Review • Assess the condition of the field and its most significant challenges and opportunities. • Identify the domains where our strategic investments can have the most impact utilizing the Road to Excellence Policy Framework as a guide. • Assess the quality and impact of the current portfolio of awards. • Identify areas of programming that deserve greater attention as well as opportunities for revision or elimination of program emphases.

17. Design of the Thematic Review • First Workshop • Identify domains where NSF’s strategic investments can have the greatest impact. • Articulate an analytic framework for organizing the portfolio that will permit an assessment of program alignment; identification of unmet needs; options for revisions of existing programs and areas of possible unnecessary duplication.

18. Design of the Thematic Review • Second Workshop • Review the portfolio and assess its quality and how well it addresses critical issues in the field. • Identify areas for future emphasis and revision/elimination. • Recommend approaches to evaluating the portfolio in the future.

19. Portfolio Review Letter Report • Evaluation of the portfolio • Assessment of the alignment of the portfolio with needs of the field • Recommendations for future action by NSF