Rising to the Occasion: Systematic STEM Academy Assessment Practices

1. Rising to the Occasion: Systematic STEM Academy Assessment Practices Virginia Department of Education Office of Career and Technical Education Services October 25, 2012 “Using Data for Program Planning and Continuous Improvement”

2. Rising to the Occasion: Systematic STEM Academy Assessment Practices Nathan Carter, Ph.D. Research Consultant

3. Background Information • In 2008, the Virginia Board of Education approved the criteria for establishing Governor’s STEM Academies. • As part of the criteria, the Virginia Department of Education (VDOE) was required to conduct annual evaluations of all established STEM Academies. • In order to evaluate STEM Academies effectively VDOE needed to: • Acquire current descriptions of each career pathway offered at a STEM Academy • Develop an informative templatethatSTEM Academies could use to complete an Annual STEM Academy Program Assessment Report

4. Typical Problems with Program Assessments • Pressure for doing a comprehensive program assessment can cause a “rush to completion syndrome” and produce results that are not very useful. • When there is not a team-oriented approach to completing a comprehensive program assessment important voices can be left out of the discussion. • In some situations comprehensive program assessment is not based on a clear understanding of core program values or agreement on what the short and long-term goals of the program are.

5. Another Common Problem: DATA OVERLOAD!!!!! If I’d known they wanted me to use all of this quantitative data – I would NEVER have asked for it!!!

6. Four General Purposes of Quality Comprehensive Program Assessment • To improve – The assessment process should provide feedback to stakeholders on how a school program can be improved. • To inform – The assessment process should inform decision makers and program stakeholders of the continual contributions and impacts of a program on meeting short and long-term goals. • To prove – The assessment process should demonstrate what the program is accomplishing and how these accomplishments are changing overtime. • To support- The assessment process should support faculty, administrators, and others as they think about what actions need to be taken to continue to support students.

7. Requirements for a Complete Assessment Process • It must be comprehensive – must cover the full range of learning outcomes that are connected to program values and goals • It must include multiple judgments – multiple sources of evidence must be used • It must include multiple dimensions – different facets of student performance must be included • It must collect direct evidence – direct measures of student attainment must be used • It must include qualitative interpretations of the data and evaluations of patterns overtime.

8. Reflect& Revise Plan AnalyzeData Implement CollectData Iterative Model of the STEM Academy Annual Assessment Process

9. Workplace Experiences Offered With Career Pathway • Job Shadowing Internship • Mentorship Cooperative Education • Student Apprenticeship Service Learning • Other STEM Academy Career Pathway Description Sheet Career Pathway Name___________________________

10. Sections of the STEM Academy Assessment Report Each section has a quantitative and qualitative data collection requirement

11. Sections of the STEM Academy Assessment Report (cont.)

12. Sections of the STEM Academy Assessment Report (cont.)

13. Data Validation Flow Chart START VDOE provided each STEM Academy with a PDF document that showed which tables and available data in the template could be populated with state-collected data. rt. STEM Academies will compare the data tables provided by the VDOE with their own data tableson the different areasteam. STEM Academies will organize their own data management team and set deadlines for populating tables in the STEM Academy Assessment Report using locally stored data sources. In the case of discrepancies , STEM Academies will use their own locally validated data to populate the tables in the Assessment Report and complete their annual evaluation. STEM Academies will submit their Annual Assessment Report and a list of any discrepancies uncovered during the data validation process to VDOE along with descriptions of career pathways END STEM Academies will create a list of all the relevant data inaccuracies between the two sources and document possible reasons for the differences.

14. Important Questions To Think About As You Move Forward How are you currently embracing a data-driven decision-making process that leads to actionable results?

15. Who is going to be a part of the program assessment team? • Consider including people from various stakeholder groups to work colloboratively on the comprehensive program assessment plan.

16. What is a realistic schedule for completing the program assessment? • When will we administer the assessment? • When will we collect the data? • When will we analyze the data? • When will we reflect upon the data? • When will we make recommendations? • When will we make the decisions about the recommendations? • When will we provide written documentation about the decisions? • When will we share the data with other stakeholders?

17. Continuing Professional Development via Video Streaming The next CTE Directors Video Streaming Session will be held November 8, 2012 Selecting Appropriate Industry Certification Examinations Presenters: Gordon Creasy Joseph Wharff

18. For Additional Information Contact George R. Willcox, CTE Coordinator Office of Career and Technical Education Virginia Department of Education P. O. Box 2120 Richmond, VA 23218-2120 804 225-2052 804 530-4560 cte.@doe.virginia.gov