Evaluation of Efforts to Broaden STEM Participation: Results from A Two-Day Workshop

1. Evaluation of Efforts to Broaden STEM Participation: Results from A Two-Day Workshop Planning Committee: Bernice Anderson Elmima Johnson Beatriz Chu Clewell Norman Fortenberry Presenters: Patricia B. Campbell & Veronica Thomas

2. Evaluation of Efforts to Broaden STEM Participation: Workshop Goals • To develop and validate a strategy by which to demonstrate the value of NSF's investment in broadening participation (BP). • To negotiate answers to two questions: • What metrics should be used for project monitoring? • What designs and indicators should be used for program evaluation?

3. Evaluation of Efforts to Broaden STEM Participation: The Workshop Report

4. The Policy Context: Historically

5. The Policy Context: Currently

6. The Policy Context: A Core Value and A Strategic Goal

7. Measuring Success: NSF – BP Programs

8. Measuring Success: Suggested Monitoring Metrics

9. Measuring Success: Recommendations

10. Broadening Participation (BP) : Critical Issues Related to Indicators and Outcomes Developing shared understanding and clarifying meaning Addressing ‘‘success” at multiple levels

11. Important Distinctions Inputs Outputs Process Outcomes

12. Inputs Resources, contributions, investments that go into the project Input indicators measure resources, contributions and investments such as: Staff Volunteers Funding Materials Facilities Investments made to support BP

13. Outputs Units of services and goods provided by the project Output indicators measure things such as the scope/size of activities, services, events, and products reaching underrepresented Numbers of students served Numbers of workshops

14. Process Ways in which project services and goods are provided Process indicators measures extent to which BP projects, programs, and strategies delivered as intended (alignment)

15. Outcomes Things project hopes to achieve; actual benefits, impact, or changes Outcomes indicators expressed in terms of changes for individuals, groups, communities, institutions, and system : Knowledge, attitude, and skill changes Behavior changes Value changes Policy, procedural, and practice changes

16. Considering BP Success at Multiple Levels Level 1: Having access to the benefits of STEM knowledge Level 2: Having access to STEM knowledge Level 3: Studying STEM Level 4: Working in STEM areas Level 5: Generating STEM knowledge

17. Problems in Determining “Success” Defining in terms of increase in absolute number or percentage Defining in terms of increase in both number and percentage Defining in terms of the end point being “parity” (absolute number)

18. Other Considerations in Defining Success Defining “parity” as a range Achieving parity, as more participate overall Considering discipline/field size to which definition of success apply Integrating qualitative indicators (e.g., broadening and transforming perspectives)

19. Other Indicators of Success Broadening Participation Individual level indicators Institutional level indicators Foundation level indicators

20. Individual (Student) Level Indicators Participation Retention, persistence, and success Experiences Attitudes

21. Institutional Level Indicators Staffing Policies, programs, and institutional commitment Accountability and rewards Monitoring, tracking, and using data for improvement Collaborations

22. Foundation Level Indicators Inclusion of information about importance of BP Review and monitoring of foundation policies/practices in terms of potential to broaden participation Diversity of professional involved with the Foundation Foundation resources devoted to BP Improvements to knowledge base about broadening participation Implementation of effective strategies at Foundation level to BP

23. Evaluating BP: Research vs. Evaluation

24. Evaluating BP: Longitudinal Tracking

25. Evaluating BP: Using Comparison Groups

26. Evaluating BP: Selecting Designs

27. Evaluating BP: Selecting Designs