For the National Science Foundation’s Advancing Informal STEM Learning (AISL) Program

1. How to Complete the Impact Section of the OPMS: Impacts, Indicators, Impact Categories, Study Designs, and Data Collection Methods For the National Science Foundation’s Advancing Informal STEM Learning (AISL) Program

2. Key Terms • AISL project impact: a broad goal that an AISL project intends to achieve with its public and/or professional audiences using a project deliverable • An impact should describe the way in which a target audience will change or benefit as a result of exposure to an AISL deliverable • AISL project indicator: used to quantify or measure the extent to which an impact has been achieved • The purpose of an indicator is to specify how you will demonstrate that an impact has been achieved How to Complete the Impacts Section of the OPMS

3. What information does the OPMS collect about impacts and indicators? • Step 1: Impacts for each deliverable • Step 2: Impact category for each impact • Step 3: Up to 5 indicators for each impact • Step 4: Study design methods that will be used to assess whether impact actually occurred • Step 5: Data collection methods that will be used to assess whether impact actually occurred How to Complete the Impacts Section of the OPMS

4. What do impacts and indicators commonly describe? • Who the target audience is that will be impacted • What the impact will address • Awareness/knowledge/understanding • Engagement/interest • Attitude • Behavior • Skills • Where/when the impact will occur/be measured • How the impact will be measured • Quantity of the target audience that will be measured How to Complete the Impacts Section of the OPMS

5. Example impact: • Intended target population • High school students who visit the exhibit will… • Type of change that will be observed • …increase their interest in… • STEM content area that is the focus of the impact • …the Earth’s moon. How to Complete the Impacts Section of the OPMS

6. Impacts and Indicators Align with Project Deliverables • An AISL project on Earth’s moon might include three deliverables for public audiences: • A museum exhibit • A website with photos, articles, and other components • A podcast series • Each of these deliverables will have one or more impacts • The examples in this PowerPoint will focus on impacts and indicators for a museum exhibit How to Complete the Impacts Section of the OPMS

7. Sample Impacts and Indicators:Exhibit on the History of Man’s Exploration of the Moon How to Complete the Impacts Section of the OPMS

8. Step 1: Identify all impacts associated with a particular deliverable • If you don’t know all of the impacts that a deliverable may accomplish, you can enter new impacts in your annual updates and final report • If you have entered impacts that a deliverable no longer addresses, you may deactivate an impact – but you will need to explain why the deliverable no longer addresses the impact • If you have unanticipated impacts, the final report asks you to describe your most important accomplishment – and any unanticipated impacts How to Complete the Impacts Section of the OPMS

9. Step 2: Determine the category for each impact Impact: Visitors will increase their awareness of the people who shaped our evolving perception of the Earth’s moon • D11a. Please indicate the appropriate category for this impact statement. (Check one.) • For definitions of the impact categories and examples, click here. • Awareness, knowledge, or understanding of a concept or topic • Engagement or interest in a concept or topic • Attitude regarding a concept or topic • Behavior regarding a concept or topic • Skills regarding a concept or topic √ Click here to learn more about Impact Categories How to Complete the Impacts Section of the OPMS

10. Step 3: Enter up to 5 indicators for each impact Impact: Visitors will increase their awareness of the people who shaped our evolving perception of the Earth’s moon D11b.What indicators will you use to determine whether this impact will be achieved? 1.  Visitors will be more likely than non-visitors to describe how an individual’s contribution shapes our perception of the moon. 2. TEXT BOX 3. TEXT BOX 4. TEXT BOX 5. TEXT BOX Click here to learn more about Indicators How to Complete the Impacts Section of the OPMS

11. What makes a strong indicator? • Indicators should be aligned with their impacts • If an impact is about knowledge, the indicator should also be about knowledge (and not behavior) • If an impact is about learning the phases of the moon, the indicator should also be about the phases of the moon (and not identifying other planets) • The best indicators are detailed, specific, and measureable How to Complete the Impacts Section of the OPMS

12. Indicators may identify more specific learning objectives than impacts • Impact: • Visitors will learn more about the moon • Indicators: • Visitors will know about the phases of the moon • Visitors will know about at least two geographic features of the moon • Visitors will be able to name at least one individual who shaped our perception of the moon How to Complete the Impacts Section of the OPMS

13. Step 4: Select each evaluation design for this impact √ √ Click here to learn more about Evaluation Designs How to Complete the Impacts Section of the OPMS

14. Step 5: Select all Applicable Data Collection Methods (note: this is a truncated list) Click here to learn more about data collection methods How to Complete the Impacts Section of the OPMS

15. If you have more questions… • Call or email your Westat contact person • Email AISLhelp@westat.com • Call or email your NSF Program Officer How to Complete the Impacts Section of the OPMS