940 likes | 1.08k Views
Improving Performance: The Five Step Process. Mimi Lufkin: Chief Executive Officer National Alliance for Partnerships in Equity California: September 2-3, 2008. The Five Step Process. Document Performance Results. GOAL: Understand the problem completely before you seek solutions.
E N D
Improving Performance: The Five Step Process Mimi Lufkin: Chief Executive Officer National Alliance for Partnerships in Equity California: September 2-3, 2008
Document Performance Results GOAL: Understand the problem completely before you seek solutions How do you analyze performance data? What questions should be addressed? What tools and methods can be used to present and analyze data? How should data quality problems be considered in analyzing data?
Perkins Accountability Measures • Participation data • 4S1 & 4P1 in Perkins III • 6S1 & 5P1 in Perkins IV # underrepresented students participating in NT CTE Participation rate: all students participating in NT CTE • Completion data • 4S2 & 4P2 in Perkins III • 6S2 & 5P2 in Perkins IV # underrepresented students completing NT CTE Completion rate: all students completing NT CTE
Data CollectionDisaggregation required in Perkins IV Special Population • Underrepresented gender students in a nontraditional CTE program • Single Parent • Displaced Homemaker • Limited English Proficiency • Individuals with a Disability • Economically Disadvantaged • Gender • Male • Female • Race/Ethnicity • American Indian or Alaskan Native • Asian or Pacific Islander • Black, non-Hispanic • Hispanic • White- non-Hispanic
California Online Data Sources • Postsecondary • CCCCO Perkins III Data • CCCCO Perkins IV Negotiation Data • Secondary • Data Quest • Cal PASS (future)
Other Data Sources • No Child Left Behind (NCLB) Report Card • National Center for Education Statistics (NCES) • District Enrollment / State Level • Other Sources???
Recommended Analyses Trends • At least 2 yrs • Prefer 3-5 yrs Comparisons • State performance level • Best performer in state • Selected peer benchmark • Set your own benchmark • Site specific • Statewide • District • School/College • Programs
Data Quality Concerns Alignment, Scope, Reliability, Time & Coverage • Alignment: Extent to which instruments accurately measure what they’re supposed to measure • How are participation #s generated? By teachers? Districts? • Scope: Breadth of measurement with respect to state-identified performance outcomes • Do districts measure single parents in programs?
Data Quality Concerns Alignment, Scope, Reliability, Time & Coverage • Reliability: Consistency in conducting measurements • Do all districts define “concentrator” similarly? • Timing: When measurement is made • Do some districts measure at diff. times of year? • Coverage: Base of students included in measurement • Include all secondary students? Only concentrators?
Perkins Consolidated Annual Report Data State District or College Program Level Analysis
Statewide Secondary and Postsecondary Participation and Completion Data
Statewide Secondary and Postsecondary Participation and Completion Data
Statewide Secondary and Postsecondary Data • What does the data say or suggest? • How has state performed in each measure? • What happened to the results over time? • What questions arise from this data?
College Perkins III Nontraditional Participation and Completion Data
College Level Data • How has the college performed on each measure? • How has the college performed as compared to the state average on each measure? • What happened to the results over time? • What questions arise from this data?
College Level Perkins III and IV Data Worksheet Complete Section A and B
State Program Participation and Completion Data09 TOP Code- Engineering and Industrial Tech
State Perkins III Nontraditional Participation and Completion Data09 TOP Code Engineering and Industrial Tech
Program Level Data • What does the data say or suggest? • How has the state performed in each measure? • What happened to the results over time? • What questions arise from this data?
College Program Participation and Completion 09 TOP Code- Engineering and Industrial Tech
College Perkins III Nontraditional Participation and Completion Data09 TOP Code Engineering and Industrial Tech
Program Level Data • What does the data say or suggest? • How has the college performed in each measure? • What happened to the results over time? • What questions arise from this data?
Program Level Perkins III Data Worksheet Complete Section C and D
Other Analyses with Perkins Data • You can do disaggregated analyses by: • Ethnicity • Special population • Suggested comparisons and analyses: • Look statewide (secondary & postsecondary) and at specific sites and in specific programs • Compare % to see if certain groups choose NTO CTE programs at higher/lower rates than their representation in CTE programs in general • Look at total numbers as well as percents. Be aware of the effect of small sample size
Completion Caveats • Completions are tracked for “concentrators” - those students who are seriously committed to the program. • There may be many dropouts before students become concentrators.
Nontraditional Participation: Special Population AnalysisEngineering and Industrial Tech (09) - Region 6 data
Worksheet Activity-case study • Read the first page of the case study • At the bottom of the sheet, you’re asked: • What performance data should they review to help them identify their problem? • Answer this through completing: • “Document Performance Results” section of “The Five Step Process Worksheet” • What data can you generate? Who would this task tell you information about? Where could you get it? • This data would help you identify initial problems about nontraditional program participation & completion
STEP TWO Identify Root Causes
Why Search for Root Causes? • Keep from fixating on the “silver bullet” strategy • Identify the conditions or factors that cause or permit a performance gap to occur • Two types of causes: • Direct cause (i.e. instructional practice) • Indirect cause (i.e. teacher training)
How to Identify Root Causes • Search for most direct & highest impact causes • Employ a systematic evidence-based process • Draw on current research and evaluation • Formulate and test theories or hypotheses • Use multiple methods and data sources • Note, you’re likely to find multiple causes
Phase 1: Identify Potential Causes • Review Research Literature • Review Program/Institutional Evaluations and Effectiveness Reviews • Conduct Focus Groups • Peer Benchmarking • Interviews & Surveys • Brainstorm
Resources on CD • Survey Instruments • Destination Success • How to Conduct Interviews • How to Conduct Focus Groups
Other Resources The New Look Self-Study Illinois Center for Specialized Professional Support
Phase 2: Analyze and Evaluate Potential Causes Group Causes Into Two Categories: • Group 1: Causes Within Your Control • School scheduling • Classroom climate • Faculty awareness and capacity • Group 2: Causes Outside Your Control • Media representation • Family demographics
Phase 3: Test and Evaluate Potential Causes Within Your Control Select root causes that: • Have the strongest theory and evidence to support them • Focus on direct causes of performance gaps • Address the most critical needs • Provide the best opportunity to have high impact on performance • Are supported by stakeholders who will help develop and implement solutions (See page 17 of the OVAE Guidebook)
Review Research Summary Root Causes Tables • Chapter 6 Tables 16, 17, 19, and 20 from, “Improving Performance on Perkins III Core Indicators: Summary of Research on Causes and Improvement Strategies” • Authors: Robert Sheets, Mimi Lufkin, and David Stevens for the National Centers for Career & Technical Education • Available at www.napequity.org
Review Research Summary Bias Literacy: A review of concepts in research on discrimination By Ruta Sevo, Ph.D. and Daryl E. Chubin, Ph.D.
Root Causes • Career Guidance Materials and Practices • Access to and Participation in STEM • Instructional Strategies • Classroom Climate • Role Models • Early Exposure
Root Causes • Curriculum Materials • Occupational Choice/Segregation • School Climate • Student Attitudes • Support Services • Student Isolation Based on Gender