Implementing and Validating a Revised TQRIS: A North Carolina Update

1. Implementing and Validating a Revised TQRIS: A North Carolina Update Edna Neal Collins Jani Kozlowski Smart Start National Conference May 1, 2013

2. QRIS Advisory Committee Met September 2009 – June 2012 Diverse group of stakeholders For-profit, non-profit, and government sponsored ECE providers State and local early education partners Analyzed the current Quality Rating & Improvement System (QRIS) with the goal of creating recommendations that will guide the evolution of the system over the next 10 years. Drafted short-term and long-term recommendations reflecting a statewide vision for quality early care and education systems and programs.  Recommendations directed to DCDEE, the Child Care Commission and other key system partners.

3. Considerations All components of a QRIS Standards Accountability and assessments Program and provider outreach & support Financial incentives linked to compliance with standards Consumer education Specific areas of emphasis Cultural competence Inclusion of children with disabilities Alignment with Foundations Systems that support the QRIS and programs

4. Advisory Committee Work Groups Ad hoc group on QRIS Structure Education and Professional Development Evaluation of the Overall System Finance Infant-Toddler Program Assessment Program Standards

5. Recommendation Process Homework Studied North Carolina Looked at research and national reports Looked at other states Listened to resource persons Extensive review of recommendations Work group ideas Analysis (who affected, resources, timeframe, RTT, etc.) Reviews by Chairs Reviews in whole group

6. QRIS Structural Considerations Raise the floor and raise the ceiling. Encourage continuous improvement. Distinguish different levels and logical “stair steps”. Allow for child care provider choice. Make it as simple as possible. Integrate work group recommendations. Provide a starting point for discussion.

7. Conceptual Framework for QRIS 1st Level (Basic Licensing Requirements) Program & Environment Ratios & Group Size Education & Professional Development POLICIES Admin Practices, Family Engagement, Cultural Responsiveness, Inclusion, PD Plans SPACE, SAFETY, COMPLIANCE, etc. Block Requirements (increase with levels) Core Requirements (maintained at all levels)

8. Conceptual Framework for QRIS Specialization Could Include CSEFEL Infant Toddler School Age STEM The Arts ETC. 2nd Level All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization) Program & Environment Ratios & Group Size Education & PD POLICIES Admin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans SPACE, SAFETY, COMPLIANCE, etc. 1st Level Basic Licensing Requirements Block Requirements (increase with levels) Core Requirements (maintained at all levels)

9. Conceptual Framework for QRIS Program & Environment Ratios & Group Size Ed/ PD Specialization Could Include CSEFEL Infant Toddler School Age STEM The Arts ETC. 3rd Level* Requirements in P&E, Ratios/Group Size, Ed/PD, PLUSSpecialization 2nd Level All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization) Program & Environment Ratios & Group Size Education & PD POLICIES Admin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans SPACE, SAFETY, COMPLIANCE, etc. 1st Level Basic Licensing Requirements Block Requirements (increase with levels) Core Requirements (maintained at all levels) * ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.

10. Conceptual Framework for QRIS 4th Level* Specialization Could Include CSEFEL Infant Toddler School Age STEM The Arts ETC. All requirements of Level 3, plus points that lead toward the Level 5 standards. Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization) Potential Programs of Distinction starting at 4th Level 3rd Level* Program & Environment Ratios & Group Size Ed/ PD Requirements in P&E, Ratios/Group Size, Ed/PD, PLUSSpecialization 2nd Level All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization) 1st Level Program & Environment Ratios & Group Size Education & PD POLICIES Admin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans SPACE, SAFETY, COMPLIANCE, etc. Basic Licensing Requirements Block Requirements (increase with levels) Core Requirements (maintained at all levels) * ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.

11. Conceptual Framework for QRIS Program & Environment Ratios & Group Size Ed/ PD Specialization Could Include CSEFEL Infant Toddler School Age STEM The Arts ETC. 5th Level* Requirements in P&E, Ratios/Group Size, Ed/PD, PLUSSpecialization Potential Programs of Distinction at the 4th and 5th Levels 4th Level* All requirements of Level 3, plus points that lead toward the Level 5 standards. Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization) 3rd Level* Program & Environment Ratios & Group Size Ed/ PD Requirements in P&E, Ratios/Group Size, Ed/PD, PLUSSpecialization 2nd Level All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization) Program & Environment Ratios & Group Size Education & PD POLICIES Admin Practices, Family Engagement, Cultural Responsiveness, Inclusion, PD Plans SPACE, SAFETY, COMPLIANCE, etc. 1st Level Basic Licensing Requirements Block Requirements (increase with levels) Core Requirements (maintained at all levels) * ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.

12. Executive Summary Process Recommendations • Consider data about how programs meet current requirements. • Map current and proposed requirements onto the proposed conceptual framework. • Collect a lot of input from stakeholders about both the requirements of the new system and how it should be implemented. • “Test” the proposed revisions using existing data and targeted data collection. • Project how programs would be distributed across a proposed new structure or structures before finalizing the revised TQRIS.

13. Implementation Plan • Executive Summary available at http://ncchildcare.dhhs.state.nc.us/PDF_forms/QRIS-Advisory-Committee-Executive-Summary.pdf • Spreadsheet of specific recommendations provided to DCDEE • DCDEE staff coordinate feedback, validation, and implementation process. • Integrate recommendations to create a coherent system. • Refer some specific recommendations directly to Child Care Commission • Refer system-level recommendations to system partners.

14. Race to the TopEarly Learning Challenge Building on the Recommendations

15. RttT-ELC • U.S. Department of Education and U.S. Department of Health & Human Services • Total federal funding of $500 million • Award Period: January 2012 – December 2015 • 9 winners: NC, MA, WA, DE, OH, MD, MN, RI, CA • NC Award: $69,991,121 • Primary focus: School readiness for children at risk

16. RttT-ELC and the QRIS • QRIS as a key component of a system to promote school readiness for all children. • QRIS-related activities in NC grant • Validating and implementing a revised TQRIS • Developing a new program quality measure for use in a TQRIS • Strengthening TA and PD related to possible QRIS quality indicators • Strengthening the workforce • Expanding the QRIS & moving programs to higher levels

17. What is TQRIS Validation? Validation of a QRIS is a multi-step process that assesses the degree to which design decisions about program quality standards and measurement strategies are resulting in accurate and meaningful ratings. Zellman, G. L. & Fiene, R. (2012). Validation of Quality Rating and Improvement Systems for Early Care and Education and School-Age Care, Research-to-Policy, Research-to-Practice Brief OPRE 2012-29. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

18. TQRIS Validation(Zellman & Fiene, 2012) • Examine the validity of underlying constructs. • Expert and empirical support • “Stakeholder validity” • Examine the measurement strategies and psychometric properties of measures. • Do they perform as expected? • Do the “cut scores” differentiate levels appropriately? • Assess the outputs of the rating process. • Relate the ratings to children’s developmental outcomes (and other important outcomes).

19. Pilot Study and Validation Study

20. Pilot/Validation Study Revised TQRIS Pilot and Validation Study (funded by RttT-ELC) • Study the impact of recommended changes. • Use results to develop blocks, points, weighting, and standards for each level. • Identify quality features that distinguish between programs at higher levels. • Relate program features and quality ratings to children’s progress. • Study the impact of.

21. Validation Study Phase 1 • Gather information from stakeholders to inform the creation of alternative TQRIS models for testing. • Generate 3-4 alternative models for testing, consistent with the Conceptual Framework, including Core Requirements, Blocks, Points, and Specializations. • Test 3-4 alternative models using existing data, targeted data collection, and additional stakeholder input. • Use results of this process to choose one model for further testing and validation in Phase II.

22. Validation Study Phase 2 (subject to approval) • Validate the proposed new Block levels. • Examine WHICH quality indicators do the best job of distinguishing among programs at the higher levels of quality. • Test reliability and validity of a few new components. • Conduct a pilot study, including assessment of relationship between quality ratings in the revised TQRIS and child outcomes. • Generate data needed to make adjustments to the revised TQRIS before implementation.

23. Recent Work • DCDEE Staff mapped individual recommendations onto conceptual framework. • QRIS Advisory Committee Work Group Chairs gave feedback. • Contract for Phase I of Validation Study awarded to FPG. • Stakeholder interviews completed. • Provider survey launched. • Focus groups in progress.

24. Challenges to Discuss • Provider buy-in • Balancing consistency and flexibility • Supporting continuous quality improvement • Supporting and recognizing the highest quality • Weaving in all aspects of quality • The role of accreditation