Presents:

1. Presents: "6 Steps for STEP Evaluation” Presented by Dr. David Blair at St. Edward's University and Dr. Raymond McGhee at SRI International. Information for participants: • Your microphone will be muted for the formal presentation. If your audio portion the presentation is not working, try running the Audio Set Up Wizard (under meeting). How to Engage: • Use the “Set Status” feature. • Type questions into the chat box. We will address them right away or as part of scheduled Q&A breaks. • Please send follow up questions to the Project Evaluator Network Working Group: http://stepcentral.net/groups/267

2. David Blair, St. Edwards University Raymond McGhee, SRI International 6 Steps to get started on your STEP Evaluation

3. Objectives • Promote an exchange of ideas on relevant evaluation topics in the STEP CoP. • Provide a general framework or roadmap for thinking about STEP evaluation tasks . • Set the stage for continued one-on-one interactions between STEP project evaluators.

4. Overview • Engage project leaders in evaluation design and development • Document project achievements • Describe challenges and adjustments made by the project • Use your different advisory committees to guide and inform your evaluation planning • Analyze data to document implementation, lessons learned and impact • Share results in a strategic fashion with key stakeholders

5. Opening Question • Who should participate in evaluation planning and why should they participate?

6. STEP 1

7. PI/Project Leader Collaboration • Communicate early and regularly with leadership • Document decisions and next steps • Show how evaluation results can be leveraged to improve implementation • Be persistent without being annoying

8. Opening Question: • What are the key research questions that your project wants to document? • Why?

9. STEP 2

10. Document Achievements • Key Activities/Number of students impacted • Data Collection • Student Characteristics • Surveys • Focus Groups • Journals • Impact Data • Other

11. Document Achievements (cont’d). • Successes Experienced • Increased GPA, retention, research interest, cohort building, student satisfaction • Project activities integrated at institutions • Off-shoots • Different versions • Other project impacts beyond intended goals • Socialization, confidence

12. Document Achievements (cont’d.) • Partnership achievements • Roles • Changes as project proceeds • Unexpected benefits • Meetings/Groups • Additional resources

13. Opening Question: • What can be learned from a project’s misfires or mistakes in implementation?

14. STEP 3

15. Describe challenges/adaptation/ assessment/sustainability • Challenges faced • Grant notification • Recruitment of participants • Funding for additional staff • Finding an evaluator • Students not taking advantage of resources • Significant changes • Document fully

16. Describe challenges/adaptation/ assessment/sustainability (cont’d.) • Intermediate measures and metrics to monitor progress • Data Analysis • Quantitative • Qualitative • Mixed Methods

17. Opening Question: • How can your advisory committee(s) best contribute to your project evaluation?

18. STEP 4

19. Advisory Board Involvement • Internal – use them to help facilitate data collection and analysis efforts as well as interpreting the results of data analyses. This board may also help frame results for different stakeholders within the participating institutions. • External – use them to help you identify key audiences for your evaluation results as well as the key messages you want to share based on your evaluation results.

20. Opening Question: • Why is it important for an evaluator to adopt "methodological pluralism" with respect to collecting and analyzing project evaluation data?

21. STEP 5

22. Lessons Learned/Overall Impact • Summary of how well project is meeting project goals • Tie back to original goals • Work with your external evaluator • Fully explain changes and corrections • Actions taken to sustain project • Additional funding • Integrating within the university culture and budget

23. Opening Question: • How can we better target our project evaluation findings so that they have the largest impact on institutional practice?

24. STEP 6

25. Dissemination of Results • Rationale – share successes to key stakeholders to secure support for additional funding and institutionalization of project efforts. Rationale can be sharpened with external advisory input.

26. Dissemination of Results (cont’d.) • Targets – with input from external advisory committee, identify multiple audiences who could use the results of your evaluation. Include targets at different levels of the institution as well as key external audiences (e.g., industry partners, sister institutions, 3rd Year Review board.) Also determine what messages, metrics, or narratives speak to the concerns of these audiences.

27. Dissemination of Results (cont’d.) • Methods – develop a dissemination plan that harnesses multiple modes of disseminating your evaluation’s results. Include print as well as electronic modes of sharing your work.

28. Dissemination of Results (cont’d.) • Products – based on the audiences identified, develop several dissemination products tailored for each audience. Collaborate PI and external advisory panel’s recommendations to determine the high value products.

29. 6 Steps to get You Started • Project leader collaboration • Document achievements • Describe challenges/adaptation/assessment/ sustainability • Advisory board involvement • Document implementation, lessons learned and impact • Dissemination of results

30. Conclusion/Questions • Future materials and resources • Suggested Checklist to help with plan preparation • General project evaluation references