Saginaw KEYS Data Analysis Training for Continuous School Improvement

1. Saginaw KEYS Data Analysis Training for Continuous School Improvement March 20 and 21, 2006Jacques Nacson jnacson@nea.org Gary Obermeyer obee@learningoptions.net

2. Workshop Agenda • Welcome and Introductions • Review of Agenda and Objectives • Your Expectations and KEYS Experiences • KEYS and Continuous School Improvement: The Action Research Cycle • KEYS Data Analysis and Interpretations--Indicators and Predictions--General Data-Related Considerations--Demo Report: Charts and Data Points--Tools and Data Analysis Worksheets--Navigating the Online Data Report and Online Resources • The NEA KEYS 2.0 Toolkit and Use of the Action and Facilitation Guides

3. Workshop Objectives • To examine KEYS data within the larger context of continuous school improvement processes • To learn how to analyze and interpret KEYS data and how results can be shared with school staff and other stakeholders • To consider strategies for how KEYS data might be used as a stimulus to drive your targeted school improvement efforts • To learn how to use KEYS support materials, including online resources, to help you take next steps beyond preliminary analysis of KEYS data

4. Problem orOpportunityIdentified Action TakingImplementation& Evaluation Data Collection Diagnosis & Refinement of Problem Action Planning Overview of Action Research Model

5. Action Research Cycle Model for Understanding and Interpreting KEYS 2.0 Data NEA 6 Keysto School Quality Process & product evaluation including effects on student achievement School completesKEYS 2.0 survey Implementation of planned intervention School receives scores on KEYS 42 indicators of school quality School makes decisions about next steps and plans for improvement Use other data, including student achievement to confirm, validate or justify Make judgments based on theories or hypotheses (if, then) to explain “why” Make inferences about our school’s quality based on the KEYS results and assessments of other data

6. General Questions and Principles to ConsiderBefore Looking at Your KEYS 2.0 Data

7. General Questions and Principles to ConsiderBefore Looking at Your KEYS 2.0 Data(continued)

8. Flow Chart of Suggested Activities/Processes For Analyzing and Interpreting Your Data School Community Completes KEYS Survey Created a School Improvement Team (SIT) ? No Yes SIT trained in action research, use of quality tools and analysis and interpretation of KEYS Results ? No Yes SIT conducted preliminary analysis and interpretation of KEYS results and prepared for involvement of whole school community ? No Yes SIT assisted whole school community to examine results and make preliminary recommendations ? No Yes SIT considered whole school’s recommendations and begins deliberative process ? No Yes SIT developed preliminary school improvement plan based on KEYS data and other data and present to school community for feedback ? No Yes ? School improvement plan approved and ready for implementation No Yes School Improvement Plan Implemented and Evaluated

9. Labels Understanding the Graphs for Each Key Vertical Axis Indicators (groups of questions that measure the same concept) KEY 1. Shared Understanding and Commitment To High GoalsRespondents Provide Direct Instruction to Students Horizontal Axis Measure of quality: 5 point scale Left side Disagree (low value) Right side Agree (high value)

10. Labels Understanding the Graphs for Each Key SchoolAverage All Schools Average Standard Deviation 90th Percentile Score • Data Points • School average (red) • All schools average (38 pilot schools) (Purple) • 90th percentile score of the pilot sample (Yellow) • Length of the horizontal bar (Purple) 1 standard deviation above and below the school average (measure of agreement) • The Goals for your school in terms of continuous improvement for each indicator • The school average moving continuously toward the right side (agree – high value of quality for that indicator) • At the same time, reduce the standard deviation (narrow the length of the horizontal bar, meaning greater agreement among respondents)

11. District Reports

13. 5.4 Safe Environment

14. Key:_________________ Constituency Group:_________________ Date of Survey:______________________ Worksheet for Analyzing KEYS 2.0 Results (Analysis of Individual Indicators)Preliminary AnalysisNOTE: THIS WORKSHEET HAS BEEN REDESIGNED Indicators Very Low (average score between 1 & 2) Low (between 2.1 & 2.6) VeryHigh (between 4.1 & 5.0) High(Narrow <1 .3pt.) Low(Wide >1.3 pt.) High (between3.4 & 4.0) Measures of Quality Degree of Consensus

15. Worksheet for Analyzing KEYS 2.0 Results (Analysis of Cross Key Indicators)(Secondary Analysis) Key:_____________________ Constituency Group:____________________ Date of Survey:________________________ Indicators Related Indicators score score score score score score score score score score score score score score score score score score score score score score score score score score score score score score score score score score score

16. NEA Resources to Support KEYS • Action Guide

17. Facilitation guide

18. Tool Box that contains a CD with all materials

22. Select group: Eastern Middle School - DEMO Return to the KEYS Homepage

25. Steps a School Might Take Once KEYS Preliminary Analyses are Completed • GAPS: Decide on one indicator or a group of indicators where gap (s) exist. • RELEVANCE: Reflect with the “team” on the relevance, importance and priority of the selection. • DATA COLLECTION-VALIDATION: Consider the need to collect additional data to validate the KEYS findings. • DATA COLLECTION-DIAGNOSIS AND REFINEMENT: Examine what is the “root” cause of the problem.

26. Steps a School Might Take Once KEYS Preliminary Analyses are Completed 5. THEORY OF ACTION: Identify and select the most appropriate solution for your context. • ACTION PLANNING: Set achievable goals and develop specific action/project plans. • IMPLEMENTATION: Action plans must be implemented for improvement to occur. • DATA COLLECTION-EVALUATION: Both process and product evaluations are necessary for learning to happen. • BACK TO STEP 1

27. Objectives for Advanced Training KEYS Data Analysis • Understand the Value of Data for Making Good Decisions • Distinguish Between Symptoms and Root Causes for Continuous Improvement • Learn How to Analyze and Use Data through the Continuous Improvement Cycle (PDCA) • Use Quality Tools to Improve Process Quality and Student Outcomes • Understand the Concept of Variation and Use Control Charts to reduce variation and improve outcomes

28. Agenda for Advanced Training KEYS Data analysis • Data and General Data Principles • W. Edward Deming – Data and Variation • Continuous Improvement Cycle (PDCA) • Quality ToolsStatistical Process Control-Quality Control ChartsGetting to the Root CauseCollecting, Organizing and Analyzing your DataSelecting the Right Intervention Based on Theory and Hypothesis TestingPlanning and Carrying Out the InterventionEvaluating the success of the Intervention