1 / 36

Measuring Performance within School Climate Transformation Grants

Measuring Performance within School Climate Transformation Grants. Rob Horner OSEP TA-Center on PBIS www.pbis.org. Goal. Define resources for measuring performance within SCTG grants that may improve efficiency, consistency and effectiveness of evaluation efforts.

uriel-mann
Download Presentation

Measuring Performance within School Climate Transformation Grants

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Performance within School Climate Transformation Grants Rob Horner OSEP TA-Center on PBIS www.pbis.org

  2. Goal • Define resources for measuring performance within SCTG grants that may improve efficiency, consistency and effectiveness of evaluation efforts. • Provide a brief summary of schools using PBIS in the U.S.

  3. Why a Framework for Multi-tiered Behavior Support? Predictable Positive Consistent Safe The fundamental purpose of a Multi-tiered Behavior Framework is to make schools more effective and equitable learning environments.

  4. All the parts Why the strategies work Logic Model Multi-tiered Intervention Strategies/Programs Build Capacity to Scale and Sustain Implementation Process Are we doing the strategy/program Fidelity of Implementation Student/ Family Outcomes Valued outcomes/ Academic, Behavior, Social Emotional

  5. Four ways the OSEP TA-Center may be helpful Aligning multiple initiatives begins by defining common fidelity and outcome measures • Coordination and Communication • Webinars and information dissemination • Intervention Content • Tier I • Tier II • Tier III • Measuring Fidelity of Multi-tiered Behavior Support • Measuring Student Outcomes

  6. Resources • Grantees are held accountable for what they submitted. • But some consistency across projects may help all evaluation efforts • Resources that may be helpful: • Evaluation Blueprint (www.pbis.org) • Evaluation Annual Reports (www.pbis.org) • State and District Capacity Assessments • www.sisep.org • Multi-tiered Fidelity Assessment (Tier I, Tier II, Tier III) • www.pbisapps.org • Multi-tiered student discipline data systems • SWIS, CICO-SWIS, ISIS-SWIS • www.swis.org, www.pbisapps.org

  7. Performance Indicators: SEA • The number of training and technical assistance events provided by the SEA School Climate Transformation Grant Program to assist LEAs in implementing a multi-tiered behavioral framework. • The number and percentage of LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that report an improvement in knowledge and understanding of the implementation of a multi-tiered behavioral framework. • The number and percentage of schools in LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that implement a multi-tiered behavioral framework.

  8. Performance Indicators: LEA • Number and percentage of schools that report an annual decrease in office disciplinary referrals. • Number and percentage of schools that report an annual improvement in the attendance rate. • Number and percentage of schools that report an annual decrease in suspensions and expulsions, including those related to possession or use of drugs or alcohol. • Number and percentage of schools annually that are implementing the multi-tiered behavioral framework with fidelity.

  9. Performance Measure Considerations • Use a Formal Evaluation Model • Measure Fidelity at All Three Tiers • Consider Measuring Improved State/ District Capacity • Measure student outcomes within a formal “decision system”

  10. Evaluation Model: Evaluation Blueprint/ www.pbis.org Multi-Tiered Behavior Support

  11. Schools Are Schools Implementing Multi-tiered Systems of Behavior Support?

  12. Possible Measures of Multi-Tiered Behavior Support Fidelity Regardless of the Intervention approach used, TA-Center Fidelity measures are available and appropriate

  13. Available from OSEP TA-Center www.pbis.org No Cost Assessors Training PowerPoint available Assessors Training Video in development TFI- Technical Adequacy (McIntosh, Massar et al. 2014) Content Validity (.95; .93; .91) Usability (12 of 14 > 80%) (15 min per Tier) Inter-rater Agreement (.95; .96; .89) Test-retest reliability (.98; .99; .99) Factor Analysis (in process)

  14. TFI Data Tier I: 2014

  15. Tier II Tier III

  16. Action Planning

  17. State/Region Capacity Does a state have the capacity to implement multi-tiered systems of behavior support? Is Technical Assistance improving the capacity of a state/region to implement multi-tiered systems of behavior support?

  18. Possible Measures of SEA/LEA Capacitywww.sisep.org • State Capacity Assessment (SCA) • Assessment of extent to which state department has capacity to implement evidence-based practices at scales of social significance? • No Cost, from OSEP’s SISEP TA-Center www.sisep.org; www.scalingup.org • Scores: Total, Sub-scale, Item • Used for initial assessment, action planning, progress monitoring • District Capacity Assessment (DCA) • Assessment of extent to which a school district (or region) has capacity to implement evidence-based practices as scales of social significance? • No Cost, from OSEP’s SISEP TA-Center • Scores: Total, Sub-scale, Item • Used for initial assessment, action planning, progress monitoring Implementers Training Video

  19. SCA/ DCA Format(Team with Coach/ Assessor) Item Data Sources Scoring Criteria Score 0 1 2

  20. Minnesota State

  21. Minnesota State

  22. Minnesota State

  23. State SCA Item Analysis: Action Planning

  24. District Capacity Is Technical Assistance improving the capacity of a district to implement multi-tiered systems of support?

  25. District Capacity

  26. District Capacity Assessment

  27. District Capacity Assessment District Capacity Assessment: Duda, et al. 2012

  28. District Capacity Assessment District Capacity Assessment: Duda, et al. 2012

  29. District Capacity Assessment

  30. District Capacity Assessment

  31. Measures of Student Benefit • Office Discipline Referrals • Tier I = SWIS • Tier II = CICO-SWIS • Tier III = ISIS-SWIS • School Climate • Under development • School Safety Survey • Risk Factor/ Protective Factor

  32. Performance Monitoring Summary • Consider including measures of MTBSF fidelity at all three tiers. • Many Tier I options • Several Tier II options • One new measure that does all three (Tiered Fidelity Inventory (TFI) • Consider assessing SEA/ LEA “Capacity to implement evidence-based interventions” • State Capacity Assessment • District Capacity Assessment • Outcome measures • Office discipline referrals (SWIS, CICO-SWIS, ISIS-SWIS)… later • School Climate • School Safety Survey Excellent Examples of State and District Reports at www.pbis.org

More Related