Evaluating Your STEM Program Using Multiple Standards Based Practices

1. Evaluating Your STEM Program Using Multiple Standards Based Practices Sam Crecelius, Diehl Evaluation & Consulting Misty Guard, Cybele Associates

2. Overview • Overview of STEM OST Programs in Indiana • IAN Specialty Standards: STEM • Dimensions of Success (DoS) • The Common Instrument • Q&A

3. Defining Afterschool STEM STEM Education: The Indiana Department of Education defines STEM Education as “an intentional, multi-disciplinary approach to teaching and learning, in which students uncover and acquire a cohesive set of concepts, competencies, and dispositions of science, technology, engineering, and mathematics that they transfer and apply in both academic and real world contexts in order to be globally competitive in the 21st Century.” Afterschool STEM: Informal STEM Education inspires student learning through hands-on, experience-based activities that enrich and add value to their school experiences. • When: Beyond school day hours, including before school, after school, intersessions, holidays, weekends, and summers. • Where: Afterschool STEM takes place beyond school day hours, in schools, community organizations, and cultural institutions such as libraries and museums.

4. Indiana STEM Landscape • Nearly 40% of afterschool programs offer STEM. • The majority of STEM programs are K-6. • The number of STEM programs has increased each year for all grade levels. • The last two 21st CCLC competitions have included priority points for STEM.

5. STEM Quality in Indiana • Varying Levels of Quality • Inquiry-Based vs. Compliance-Based • Hands On & Minds On • Evaluation to Drive Professional Development

6. IAN Standards • IAN standards outline the path and steps to achieve high quality youth OST programs. • STEM standards: • Taskforce of STEM experts in education, business, and youth development • Based on national standards and best practices • Indiana leading in development of OST STEM standards.

7. Informal STEM Education • Inspires student learning through hands-on, experience-based activities. • Enriches and adds value to formal school experiences. • Outside of school day hours • Provided by school systems, community organizations, and cultural institutions

8. Youth Outcomes for STEM Programs • Increased interest and excitement in STEM learning. • Increased STEM skills, understanding, knowledge, and competence. • Increased awareness and interest in STEM education and career pathways.

9. IAN Specialty Standards: STEM • Program Culture & Environment • Staff Development & Expectations • Program Focus • Data Collection and Impact • Scale: Exceeds Standards, Meets Standards, Approaching Standard, Standard Not Met, Don’t Know/NA

10. Program Culture & Environment • The program creates an inspiring STEM learning environment for all youth. • STEM resources, equipment and supplies support STEM learning. • The STEM program engages families, schools, and community.

11. Staff Development & Expectations • Staff receive professional development that increases their confidence and ability to facilitate quality STEM learning.

12. Program Focus • Activities inspire and engage youth in STEM learning. • The program provides opportunities for youth to learn STEM content, skills, and knowledge.

13. Data Collection & Impact • The program utilizes a variety of data to measure the impact of its STEM programming.

14. Dimensions of Success (DoS) • Validated Afterschool STEM Observation Tool • Developed by Harvard’s Program in Education, Afterschool and Resiliency (PEAR) • Assesses 12 Dimensions of Afterschool STEM Quality • Appropriate for Multiple Afterschool Settings

15. Dimensions of Success (DoS)

16. Structure of Visits • Two visits will be conducted during the year. • Observer will observe a single activity in its entirety (or up to 2 hours). • The observer records a detailed account of activity. • Observer uses the evidence to score each dimension. • Detailed results (scores, evidence, and feedback/recommendations) are provided following the visit.

17. Common Instrument: Development • Reliable survey tool measuring engagement and interest in informal science and OST programs. • Key features: • Easy to administer quickly • Used in a large variety of settings • Concise for a variety of practitioners across various disciplines to administer to young people. • Provides: • Baseline information regarding youth enrolled in STEM programs and activities. • Allow tracking of longitudinal effects • Used for pre-post analysis

18. Common Instrument • Purpose: • Role of informal environments and OST to spark and retain student interest and engagement in STEM. • Assess child and adolescent interest and engagement in science. • Youth 10 years and older • 16 self-report items • Retrospective and Pre/Post options

19. Benefits of the Common Instrument Evaluation Program Quality Improvement • Program Quality Improvement: • Multi-step process identifying areas of successes and challenges. • Includes Evaluation • Evaluation: • Systematic collection and analysis of information about program activities, characteristics and outcomes to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. • Must be done INTENTIONALLY.

20. Questions

21. Contact Us IAN STEM Evaluation Committee STEMquality@indianaafterschool.org (317) 822-8211 www.indianaafterschool.org Sam Crecelius Misty Guard Diehl Evaluation & Consulting Services Cybele Associates sam@diehlconsulting.orgcybeleassoc@gmail.com www.diehlconsulting.orgwww.cybeleassociates.com 812-434-6753 678-740-3694