Evaluation Projects: Maximizing Impact of Computing Corps Activities

1. Evaluation Projects STARS Evaluation Assistant Webinar 1 September 19, 2014

2. Part 1: What Evaluation Assistants Do

3. What is Evaluation?

4. How do we tell our story?

5. EA Role What you do How you do it • Team Lead for Local Evaluation: • Activity Reports (1 per activity) • Story Teller of Your STARS Computing Corps • Narrative Reports (April 15) • Applied Researcher • Evaluation Project and poster at Celebration 2015 • Submit Activity Reports Online ongoing • Participate in meetings and webinars monthly • EA ToolkitSupport Resources Demonstration

6. Recap of EA Deliverables FALL 2014 Spring 2015 • Activity Reports • Consent Forms & Registration • Fall Synopsis of Activity Reports by Thanksgiving • EA Project Milestones: • Logic Model (Oct 3) • Timeline (Oct 24) • Activity Reports • Narrative Report (April 15) • Evaluation Poster (May 1)

7. Part 2:Understanding Evaluation Evaluation Projects

8. Evaluation is…. • Evaluation is the systematic assessment of the worth or merit of some object (Trochim) • Applied social science research • A way of viewing the project, its participants, it effect • Accounting for CONTEXT and MEANING • Determining the VALUE of something • Measurements are the currency

9. Understanding Evaluation 1.Systematically define what data will be collected, how, when & by whom 2.Process the data to determine what it means within specified contexts 3.Tell constituents about findings in appealing & appropriate ways 4.Disseminate broader findings to share within disciplines

10. Evaluative Thinking Reflective practice of systematically using data to inform decisions Ongoing process Approach to making decisions, what actions to take, to become/remain effective at the organizational (or individual) goals It is LEARNING and APPLYING learning

11. Academic Research Evaluation and Academic Research Evaluation • Applies research methods to answer questions about a program’s efficacy & value • Considers context (stakeholders, climate, etc.) • Makes a judgment call on WORTH, and defines value structure (i.e. value to whom?) • Applies research methods to answer a research question • Knowledge gained in scientific discipline • Does not apply value structure; values are seen as bias

12. In summary, evaluation… Uses research methods Applies methods to social, political and economic realities Helps to interpret academic research within larger socio-economic/geo-political realities Answers the “so what?” of observed outcomes Research = Knowledge Discovery Evaluation = Extracting Meaning from Knowledge

13. STARS Alliance Logic Model Context Goals Inputs Outputs Outcomes/Impact Computing enrollment & graduation is in a state of decline nationally K12 students lack appropriate background to pursue computing Computing pipeline lacks diversity K16 Recruitment Graduate School Bridging Post-secondary Student Retention Scholarship Advancement Alliance Sustainability Dissemination Scaling NSF Funding for 50 Alliance members K12 Partnerships STARS Leadership Corp Faculty Associates Community & Industry Partners K16 Outreach Celebration Demonstration Projects Social media Online Repository & Toolkit Dissemination Evaluation K16: Develop, test, deploy academic interventions; Pedagogical reform Increased post-secondary enrollment & graduation Increased scholarship & research Increased dissemination of best practices & national adoption of Alliance Community of Practice Increased BPC partnerships & advocacy Formative assessment: SLC and K16 Outreach, Partnerships, Tools, Celebration Community of Practice & National Resource: Lessons Learned, Best Practices, Social Network, Resources & Evaluation Repository/Toolkit, Research, Summative Evaluation

14. Logic Model Framework for Computing Corps Outcomes/Impact Context Goals Inputs CHALLENGES PROBLEM TO SOLVE What do you hope to accomplish and how will you know? ACTIVITIES RESOURCES Short & Long Term Accomplishments E.G. Tools/relationships built, developed Numbers served Attitudes Actions Due Oct. 3

15. Overview of EA Projects • Evaluation project that the EA manages • Can do alone or manage a team of Corps students at your school • A distinctive measurement of the impact of a Corp project at your school • Example- measure impact of high school outreach program on high school student interest in computing • Example- assess mentoring program at your school

16. EA Projects • Are NOT a description of what the Corp is doing or has done • Are a summation of data collected, observed, documented about the impact of what the Corp has done at a school

17. Components of EA Projects • Determine which Corp program to assess at your school • Align project to the program goals • E.g. if your Corp conducts high school outreach, what are the goals? • Determine which goal(s) to measure and what data to collect • Complete a project description and timeline (see www.starsalliance.org EA Toolkit for samples) • Assess the program and report findings via Celebration Poster Session

18. Get Support • Corps Project Teams • Share ideas and tools within your STARS Computing Corp • Ask prior EAs at your school for input, ideas, suggestions • Visit the EA Toolkit • Sample IRBs, sample outreach surveys and more • Discuss with EA community • Online or via monthly meetings

19. Q&A Evaluation Projects