MIECHV Benchmark Data Collection and Evaluation

1. MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public Affairs University of Illinois

2. Overview • What are the benchmarks and why are they necessary? • Federal requirements • Evaluation and data collection • Performance measures • Continuous quality improvement (CQI)

3. Purpose of Presentation • Increase understanding for role of Benchmarks, Benchmark Reporting Requirements • Understand the reporting and data collection requirements for the MIECHV programs • Identify the roles that each organization involved in Benchmark reporting will play, and what additional roles that CPRD can play for MIECHV and home visiting services • Identify key performance measures for MIECHV programs, both for Benchmarks and continuous quality improvement efforts • Understand the role of the field data collectors

4. And what is not the purpose of this presentation?

5. What is MIECHV? Maternal, Infant, Early Childhood Home Visiting Over time, families and home visitors build partnerships and work together to: • Improve health and development. • Prevent child injuries, child abuse, neglect, or maltreatment, and reduce emergency department visits. • Improve school readiness and achievement. • Reduce crime, including domestic violence. • Improve family economic self-sufficiency. • Improve the coordination and referrals for other community resources and supports.

6. How do we define success? • Providing high-quality home visiting services that positively impact the lives of children and families enrolled in the MIECHV programs • Meeting and exceeding Benchmarks • Findings ways to pool our knowledge collectively to improve statewide delivery of home visiting services.

7. How does MIECHV fit in? • MIECHV is designed to expand HV services into priority populations: • Reside in communities in need of services & have low income • Include pregnant women who have not attained the age of 21 • Have a history of child abuse • Have a history of substance abuse • Have users of tobacco products • Have a history of, or have children with, low student achievement • Have children with developmental delays or disabilities • Include members of the military

8. MIECHV Priority Communities • Rockford- Winnebago County • 65.1% White, 20.5% African American, 15.8% Hispanic • Vermilion County • 83.6% White, 13.3% African American, 4.4% Hispanic • Chicago- Cook County • 45% White, 32.9% African American, 28.9% Hispanic • Elgin City ( border of Cook and Kane Counties) • 69.5% White, 7.4% African American, 43.6% Hispanic • Cicero Town- Chicago • 51.9% White, 3.8% African American, 86.6% Hispanic • Macon County • 79.8% White, 16.6% African American, 2.0% Hispanic Doula Sites: N. Lawndale, E. Garfield Park, Vermilion County, Rock Island County and Waukegan

9. The program goals, per the HRSA website: • Strengthen and improve the programs and activities • Improve coordination of services for at-risk communities • Identify and provide comprehensive services to improve outcomes for families who reside in at-risk communities

10. MIECHV Benchmarks • There are a total of 6 Benchmark areas • These reflect overall domains in which we expect home visiting services to have an impact. • Each benchmark has a set of constructs, and these total 37 constructs across the 6 benchmarks • Benchmarks were developed in the Legislation, and the constructs were developed by HRSA to address the benchmarks.

11. Why do we have Benchmarks? The legislation funding MIECHV required quantifiable, measurable improvements for the populations receiving services. Programs must demonstrate improvement in the following Benchmark areas: • Improved maternal and newborn health • Prevention of child injuries, child abuse, neglect, or maltreatment, and reduction of emergency department visits • Improvement in school readiness and achievement • Reduction in crime or domestic violence • Improvements in family economic self-sufficiency • Improvements in the coordination and referrals for other community resources and supports

12. Where do the Benchmarks come from?

13. Now, let’s look at the Benchmarks individually • The Benchmarks contain constructs, specific measures to address each concept • Reflect outcomes of home visiting services

14. Improved Maternal and Newborn Health • Constructs: • Prenatal Care • Prenatal use of alcohol, tobacco, or illicit substances • Postpartum use of contraception and Interpartum Interval • Screen for Maternal Depressive Symptoms • Duration of breastfeeding, well-child visits, and maternal health insurance coverage

15. 2. Prevention of Child Injuries, Child Abuse or Maltreatment and Reduction of Emergency Department Visits Constructs: • Visits for child and mother to the emergency department from all causes • Information provided or training of participants on prevention of child injuries • Incidence of child injuries requiring medical treatment • Reports of suspected maltreatment for children in the program, reported substantiated maltreatment, and first time victims of maltreatment

16. 3. Improvements in School Readiness and Achievement Constructs: • Parent support and knowledge for children’s learning and development • Parenting behaviors and parent-child relationship • Parent emotional well-being or parenting stress • Child’s communication, language and emergent literacy, cognitive skills • Child’s positive approaches to learning and attention • Child’s social behavior, emotional regulation and emotional well-being • Child’s physical health and development

17. 4. Domestic Violence Constructs: • Screening for domestic violence • Of families identified for the presence of domestic violence, number of referrals made to relevant domestic violence services

18. 5. Family Economic Self-Sufficiency Constructs: • Household income and benefits • Employment or education of adult (mother and father) members of the household • Health Insurance Status

19. 6. Coordination and Referrals for Other Community Resources and Supports Constructs: • Number of families identified and referred to available community resources • The number of families who complete referrals to available community resources • Number of agencies with which the home visiting provider has a clear point of contact in the collaborating community agency • Number of agencies with which the home visiting provider has established a formal memorandum of understanding

20. What the Evaluation is and is not • The Evaluation is: • A way to measure and address the challenges in scale-up implementation of HV services • A way to show Congress that tax dollars spent on additional HV services is important and useful • A way to introduce statewide continuous quality improvement efforts • A goal setting mechanism • The Evaluation is not: • A performance assessment to be used for funding decisions • A method to weed out the poor performing HV sites • A way to collapse the variety of model HV programs into a single government-managed HV organization

21. Illinois MIECHV: A Partnership Approach CPRD is involved in the following areas: • In partnership with the state, developing measurement procedures and systems which adequately address the Benchmark reports • Coordinating Benchmark data collection with state and sites • Work together with home visitors to gather benchmark data • Query data from MIECHV data system and prepare reporting to the feds annually • Utilize data to track individuals through time in services • Collect home visiting services satisfaction information • Help clarify possible barriers to engagement • Lead Home Visiting Continuous Quality Improvement efforts in MIECHV and throughout the Home Visiting State System

22. What if we don’t make improvements on the benchmarks after 3 years? • Would need to develop a plan of action • CPRD, the Governor’s Office, and individual community programs would work together to develop a plan and next steps • The CQI process is crucial to next steps

23. What is CQI and how will CPRD aid in the CQI efforts? I can tell you with 95% confidence that there is less than a 65.6% possibility that the CQI will simply generate 34.8% more meaningless statistics.

24. What are the cohorts mentioned in the Benchmarks? • The term “cohort” refers to the idea of a group of families who enter services together. • But families are entering and exiting HV services all the time • Cohorts in MIECHV follow one group of people and compare them to another group enrolled a year later • Cohorts help us measure program change and effectiveness over time • How will we know if individuals are changing? • Individual change is reviewed in CQI process

25. Who are Field Data Collectors? • What does it mean to be a field data collector? • How will we all stay in contact? • How will we ensure regular communication and updates? • Working with the home visiting sites and home visitor • Will use a shared calendar through Google Calendar (E.g. 'macondc@gmail.com‘), e-mail, call and text.

26. The Role of Field Data Collectors in the Benchmarks • Some Measures require rigorous approach for reliability • The Benchmarks created a large and taxing load on home visitors • Some benchmarks require that the parent not be hindered by watching their child, such as needing to fill out a longer assessment

27. On-site Data Collection • Field data collectors will work with home visitors to collect measures from families • Assess the families at three time periods • 2-4 weeks after enrollment • 1 year and 2 year follow-ups • Informed Consent – depending on age or emancipation status • Parental consent – if mother is less than 18 years old, written parental/guardian consent is required and self-assent • Self-consent – emancipated or 18 years or older, written assent is required • Gift cards are provided to participants

28. What is the PICCOLO? Parenting Interactions with Children: Checklist of Observations Linked to Outcomes • PICCOLO items describe positive parenting • PICCOLO helps practitioners identify parenting strengths to encourage parents to do more

29. What does PICCOLO measure? From the research literature, there are identified four important kinds of “developmental parenting” that support child development: • Affection • Responsiveness • Encouragement • Teaching

30. What does PICCOLO look like?

31. 29THINGS PARENTS DO THAT PREDICT SCHOOL READINESS • Speak warmly • Smile at child • Praise child • Stay physically close to child • Say positive things to child • Interact in positive ways with child • Show emotional warmth • Pay attention to what child is doing • Change activities to meet child’s interests or needs • Be flexible when child changes interests • Follow what child is trying to do • Respond to child’s emotions • Look at child when child talks or makes sounds • Reply to child’s words or sounds • Wait for child’s response after making a suggestion • Encourage child to do things with toys • Support child’s choices • Help child do things on his or her own • Verbally encourage child’s efforts • Offer suggestions to help child • Show enthusiasm about what child does • Explain reasons for something to child • Suggest activities to build on what child is doing • Repeat or expand child’s words or sounds • Label objects or actions for child • Engage in pretend play with child • Do activities in a sequence of steps • Talk about characteristics of objects • Ask child for information

32. What is the KIDI? Knowledge of Infant Development Inventory • The theoretical base for the KIDI was the theory that parental knowledge of infant development can influence parental behavior and child rearing practices. • The KIDI is a 58-item instrument that was designed to obtain comprehensive information on parents’ factual knowledge of parental practices, child developmental processes, and infant norms of behavior. • The KIDI is designed to be easily accessible to persons with limited education and to be culturally neutral.

33. What is the PSI? The Parenting Stress Index - Short Form • The PSI-SF consists of 36 items derived from the PSI which comprise three scales: Parental Distress, Difficult Child Characteristics, and Dysfunctional Parent-Child Interaction. • Designed to evaluate the magnitude of stress in the parent-child system • The PSI-SF is commonly used as a screening and triage measure for evaluating the parenting system and identifying issues that may lead to problems in the child's or parent's behavior. • Asking parents to complete the PSI—SF can be uncomfortable. Field Data Collectors explain that it will provide information that will help identify the sources and different types of stress that every parent can experience.

34. What is the HOME? • Home Observation for Measurement of the Environment (HOME) is a descriptive profile which yields a systematic assessment of the caring environment in which the child is reared. • The primary goal of the instrument is to measure, within a naturalistic context, the quality and quantity of stimulation and support available to a child in the home environment. • Its focus is on the experience of the child in the home environment.

35. Other Resources • Visit Tracker has various recorded webinars available. New staff should watch Illinois MIECHV Training for New Staff. https://visittrackerweb.com/ • The Governor’s Office webpage has numerous up-to-date documents posted including a MIECHV Data Collection Manual. http://www2.illinois.gov/gov/OECD/Pages/MIECHVP.aspx • Field Data Collection brochures for each MIECHV community (contact Mary Anne Wilson for copies) • Contacts • Teresa Kelly, Project Director, Strong Foundations Partnership- Governor's Office of Early Childhood Development 312-814-0905 teresa.m.kelly@illinois.gov • Lesley Schwartz, Manager of Program Evaluation- Governor's Office of Early Childhood Development 312-814-4841 lesley.schwartz@illinois.gov • Joanna Su, Community Systems &Capacity Building Manager-Governor's Office of Early Childhood Development 312-814-6741 Joanna.su@illinois.gov • Audrey Moy, Manager of Compliance- Governor's Office of Early Childhood Development 312-814-3141 audreymoy@illinois.gov • Peter Mulhall, Director , CPRD-University of Illinois 217-244-3231 mulhall@illinois.edu • Mary Anne Wilson, MIECHV Research Project Specialistat CPRD, University of Illinois 217-333-3231 mawilso@uillinois.edu • Deborah Kemmerer, MIECHV CQI Specialist at CPRD, University of Illinois 217-300-4158 dkem@uillinois.edu