Findings from Westat's Evaluation of the State Improvement Grant Program: Implementation Issues, Systemic Change, and Pr

1. Findings from Westat’s Evaluation of the State Improvement Grant Program:Implementation Issues, Systemic Change, and Project Outcomes Tom Fiore, Westat Alison Langham, Westat Larry Magliocca, Emeritus, The Ohio State University Presentation to the SIGnetwork SPDG Evaluators’ Session February 11, 2010 Project Funded by: Office of Special Education Programs U.S. Department of Education

2. Presentation Outline • Introduction (Tom) • Project implementation and leadership (Alison) • Systemic change inquiry (Larry) • Outcomes (Tom) Westat

3. Background on the SIG Program Evaluation • Commissioned by OSEP in 2000 • Five-year cooperative agreement, that continued for six years • Formative • Focused on the overall SIG Program, not individual states—but individual SIG projects were the unit of analysis • Not intended to provide technical assistance to projects • Instead, intended to describe what states were doing to inform OSEP, and the states themselves Westat

4. Background on the SIG Program Evaluation • Premises of the SIG Program that guided our evaluation • Systems change is necessary to effect significant statewide improvements • Improving systems requires comprehensive planning that involves multiple individuals, agencies, and institutions • SEAs need to play a leadership role in engaging partners and bringing about the systems changes Westat

5. ALISON Westat

6. Project Implementation and Leadership • SEAs need to play a leadership role in engaging partners and bringing about the systems changes • When the SIG Program began, most SEAs did not have previous experience designing, implementing, or evaluating a project of this nature. • We focused early in our evaluation on issues of implementation and came back to it later. Westat

7. Project Implementation and Leadership • Every SIG project was implemented somewhat differently • We didn’t have NIRN Westat

8. Project Implementation and Leadership • Confirmation of importance • Corroboration of our data Westat

9. Project Implementation and Leadership Environmental Changes Feedforward F K Needs Assessment and Prioritization Functions Vision and Purpose-Setting Functions Planning and Decision-Making Functions Implemen- tation Functions • Evaluation Functions • Embedded Inquiry • Formal Evaluation B C D E Outcome State I. Improved System Function II. Scaling- Up of Practices III. Improved Student Outcomes A J Unique Contextual Factors Feedback 1 G I Worth Feedback 2 Reflec- tion H Westat

10. tomfiore@westat.com Westat 10

11. Project Implementation and Leadership Westat

12. Project Implementation and Leadership • Influence • Persuasion • Guidance • Authority • Use of position or title • Control of funds • Accountability • Setting expectations • Establishing standards of performance Westat

13. Project Implementation and Leadership • The Administrative Leadership Study • Nine states • Iterative data collection and analysis • Qualitative coding Westat

14. Project Implementation and Leadership • Influence Strategies Strategy A: Implementing professional development (PD) as a systemic tool for change Strategy B: Communicating a vision of change Strategy C: Creating local buy-in Strategy D: Integrating PD efforts with general education reform Strategy E: Using collaborative working partnerships that promote joint learning Westat

15. Project Implementation and Leadership • Authority Strategies Strategy F: Asserting delegated authority Strategy G: Setting direction Strategy H: Justifying actions by evoking a higher authority Strategy I: Providing support for partners and subgrantees who buy-in Westat

16. Project Implementation and Leadership • Accountability Strategies Strategy J: Standardizing the evaluation methodology Strategy K:Prioritizing the tasks associated with accountability Strategy L:Using accountability data to justify changes in project Westat

17. Project Implementation and Leadership • Administrative Leadership Study findings • Every state used multiple leadership strategies • States used influence strategies most and accountability strategies least • Operational context mattered in the leadership strategies states chose Westat

18. LARRY Westat

19. Systemic Evaluation Inquiry • Conducting systemic change inquiry • How logic and systemic models differ • Logic Models provide a representation of the linear process of the theory of action • Systemic Inquiry Model provides a discovery (non-linear) process model of the theory of change Westat

20. Systemic Evaluation Inquiry • How logic and systemic models differ • Logic Models provide a representation of the linear process of the theory of action • Systemic Inquiry Model provides a discovery (non-linear) process model of the theory of change Westat

21. Systemic Evaluation Inquiry Model Environmental Changes Feedforward F K Needs Assessment and Prioritization Functions Vision and Purpose-Setting Functions Planning and Decision-Making Functions Implemen- tation Functions • Evaluation Functions • Embedded Inquiry • Formal Evaluation B C D E Outcome State I. Improved System Function II. Scaling- Up of Practices III. Improved Student Outcomes A J Unique Contextual Factors Feedback 1 G I Worth Feedback 2 Reflec- tion H Westat

22. Systemic Evaluation Inquiry • Purpose of the model • The Systemic Evaluation Inquiry Model enabled the evaluation team to understand each state's systemic change efforts through the SIG by • identifying important evaluation questions • discovering effective strategies of systemic change • comparing those strategies across states and outcomes Westat

23. Systemic Evaluation Inquiry • Discovering how systemic change occurs • Once the Inquiry Model was tested and validated as suitable, the evaluation questions (indicated by letters in the Inquiry Model) served as our data gathering mechanism through classic formative evaluation methodologies with each state Westat

24. Systemic Evaluation Inquiry • Focus on planning and decision-making function • Generic function required by each SIG • Development of Key Evaluation Questions • Testing the model: Does it capture the implementation process? • Valid model: How and what strategies are being used to implement? • How feedback and feed-forward mechanisms are being used? Westat

25. TOM Westat

26. SIG Evaluation’s Focus on Outcomes Goals Strategies & Activities Outputs Direct Outcomes Intermediate Outcomes Long-Term Outcomes Westat

27. SIG Evaluation’s Measurement of Individual Project Outcomes • Assessed outcomes, as reported by SIG projects • Number, type, and quality of outcomes • Focused on the first round of SIG funding, which went to 48 states • Focused on intermediate and long-term outcomes • Emphasized the distinction between outputs and outcomes Westat

28. SIG Evaluation’s Measurement of Individual Project Outcomes • Outputs are • Implementation of planned activities • Project accomplishments • Direct results of the SIG project activities • Project products and programs as well as the customer contacts resulting from them Westat

29. SIG Evaluation’s Measurement of Individual Project Outcomes • Short-term outcomes are what customers do or become as a result of outputs—usually changes in the customers’ actions or behaviors based on knowledge or skills acquired • Intermediate outcomes are the result of the changed actions or behaviors of the customers that, usually, have a direct impact on students • Long-term outcomes are broadest outcomes, usually the child outcomes of ultimate interest Westat

30. SIG Evaluation’s Measurement of Individual Project Outcomes • Activities/outputs of interest for our measure of SIG project outcomes • Training of professional personnel, paraprofessionals, parents, and administrators • Technical assistance • Dissemination of materials Westat

31. SIG Evaluation’s Measurement of Individual Project Outcomes • Identified four categories of intermediate and long-term outcomes: • Improved child performance • Changes in adult behavior • Improved systems functioning • Scaling up of existing successful practices Westat

32. Improved Child Performance (Long-Term Outcome) Changes in Adult Behavior (Intermediate Outcome) Improved System Functioning (Intermediate Outcome) Scaling Up of Existing Successful Practices (Intermediate Outcome) SIG Evaluation’s Measurement of Individual Project Outcomes Westat

33. SIG Evaluation’s Measurement of Individual Project Outcomes • What counted as improved child behavior outcomes • Improved test scores • Educator or parent reports or observations of specific improvements in child learning or behavior • Decreased referrals to special ed or increased return to general ed • Reduced suspension or referral rates • Improved graduation or dropout rates • Increased post-secondary job or education placement rates Westat

34. SIG Evaluation’s Measurement of Individual Project Outcomes • What did NOT count as improved child behavior outcomes • Test scores not linked to a specific initiative • General statements of improved learning or behavior Westat

35. SIG Evaluation’s Measurement of Individual Project Outcomes • What counted as improved adult behavior outcomes • Increased teacher retention rate • Direct observations of use of new skills • Self-reports of use of new skills Westat

36. SIG Evaluation’s Measurement of Individual Project Outcomes • What did NOT count as improved adult behavior outcomes • Number of adults trained • Adult satisfaction with training • Adult self-reports that they plan to use what they learned in training Westat

37. SIG Evaluation’s Measurement of Individual Project Outcomes • What counted as improved systems functioning outcomes • Self-reports of increased collaboration • Improved organizational structures • Improved staffing patterns • New and potentially effective curriculum, materials, data sources, laws, or accountability schemes • Increased test participation [more] Westat

38. SIG Evaluation’s Measurement of Individual Project Outcomes • What counted as improved systems functioning outcomes [cont.] • A greater number or percentage of fully certified teachers or administrators • Evidence of organizational learning from previous efforts • Improved responsiveness of existing systems • Increased inclusion • Increased effective use of data Westat

39. SIG Evaluation’s Measurement of Individual Project Outcomes • What did NOT count as improved systems functioning outcomes • Organizational agreements signed or commitments made • Number of trainings that occurred or number of participants in training • Meetings attended • Subgrants awarded • Stipends awarded Westat

40. SIG Evaluation’s Measurement of Individual Project Outcomes • What counted as scaling up of successful practice outcomes • Evidence of broad implementation of new practices • Outcomes from a broad range of sites • Evidence of sustainability of an initiative once propagated Westat

41. SIG Evaluation’s Measurement of Individual Project Outcomes • What did NOT count as scaling up of successful practiceoutcomes • Plans or commitments to expand an initiative • Number or locations of trainings • Subgrants awarded Westat

42. SIG Evaluation’s Measurement of Individual Project Outcomes • For purposes of validity and reliability, after extracting outcomes from SIG project documents, the evaluation team conducted a series of telephone conversations with SIG project directors and staff • Confirm that all SIG project outcomes had been reported in SIG project documents and accurately extracted • Confirm that all of the outcome statements in SIG documents were, in fact, SIG project outcomes Westat

43. SIG Evaluation’s Measurement of Individual Project Outcomes • Standards in four areas used to judge outcomes found in SIG project documents • EvidenceStandard—qualitative or quantitative data shows that the described outcomes really happened • SIG Link Standard—the SIG project caused the outcome • Quality Standard—the described outcome helps children with disabilities • Impact Standard—the described outcome has had or has the potential to have a sustainable statewide impact Westat

44. SIG Evaluation’s Measurement of Individual Project Outcomes • A set of evaluation criteria were employed to determine the overall value of each reported outcome—that is, to rate each outcome that qualified as an outcome for the purposes of the SIG Program evaluation • Two members of the evaluation team independently assigned a set of numeric scores to each identified SIG project outcome. Westat

45. SIG Evaluation’s Measurement of Individual Project Outcomes • EvidenceStandard—Does qualitative or quantitative data show that the described outcomes really happened? 3: Fairly strong evidence 2: Marginally sound evidence 1: Evidence not verifiably sound 0: No evidence Westat

46. SIG Evaluation’s Measurement of Individual Project Outcomes • SIG Link Standard—Did the SIG project cause the outcome? 3: Definitely a result of the SIG 2: Definitely SIG-influenced or plausibly a result of the SIG 1: Plausibly SIG-influenced 0: No evidence of SIG influence Westat

47. SIG Evaluation’s Measurement of Individual Project Outcomes • Quality Standard—Will the described outcome help children with disabilities? 3: Clearly a positive change in outcomes for children with disabilities 2: Likely to represent or lead to a positive change in outcomes for children with disabilities 1: Might represent or lead to a positive change in outcomes for children with disabilities 0: No evidence change will improve outcomes for children with disabilities Westat

48. SIG Evaluation’s Measurement of Individual Project Outcomes • Impact Standard—Will the described outcome have a sustainable statewide impact? 3: Sustainable statewide impact already achieved 2: Evidence for substantial statewide impact now or eventual sustainable statewide impact 1: Demonstration only or limited scale-up 0: No evidence of impact Westat

49. SIG Evaluation’s Measurement of Individual Project Outcomes • Overall outcome scores are the product of 0 – 3 scores on each of the four standards • Product was used instead of the sum because each standard is presumed to be essential—strength on one standard does not fully counterbalance weakness on another • Zero in any standards nullifies the entire score Westat

50. SIG Evaluation’s Measurement of Individual Project Outcomes • The overall strength of each non-zero outcome was classified as robust, solid, probable, and possible Westat