SBHCs Academic Outcomes: Are We Asking the Right Questions

1. SBHCs & Academic Outcomes: Are We Asking the Right Questions? Gorette Amaral Mona E. Mansour Sara Ann Peterson Susan Russell Walters

2. Why we decided to explore this research road�. We know that School Based Health Centers (SBHCs) provide a convenient, accessible source of valuable medical and mental health services on school grounds -- where students may need them most in order to succeed academically. As evaluators, we are increasingly asked to demonstrate the impact of SBHCs on academic outcomes. After lengthy conversations with our colleagues nationally and locally, we decided that it would be wise to see what had already been documented in the literature� We know that School Based Health Centers (SBHCs) provide a convenient, accessible source of valuable medical and mental health services on school grounds -- where students may need them most in order to succeed academically. As evaluators, we are increasingly asked to demonstrate the impact of SBHCs on academic outcomes. After lengthy conversations with our colleagues nationally and locally, we decided that it would be wise to see what had already been documented in the literature�

3. First, What Does the Literature Say?

4. Literature Review To see how academics are impacted by SBHCs, we searched for� Peer-reviewed journal articles, evaluation studies and reports Written by private and public research and government organizations In health, education, and psychology fields

5. A total of eight studies that examined the relationship between SBHCs and academic outcomes through the use of an experimental study were reviewed. IS WEBBER A total of eight studies that examined the relationship between SBHCs and academic outcomes through the use of an experimental study were reviewed. IS WEBBER

6. Studies found a Positive Impact from SBHCs on� Absences 4 (of 7) Promotion to the next grade 2 (of 2) Withdrawal/drop out rates 2 (of 2) Disciplinary problems 2 (of 3) Failing grade 1 (of 2) Tardiness 1 (of 1) Seven studies examined absence and/or attendance as an outcome. Four of the studies found a positive relationship between SBHC use and fewer absences, while three of the studies found no relationship between SBHC use and absences. Three studies looked at the relationship between SBHC use and disciplinary outcomes. Two of the studies found a positive relationship between either the use and/or the presence of a SBHC with fewer disciplinary problems, while one study found no relationship. Two studies examined promotion to the next grade and two analyzed withdrawal/drop out rates; all of these studies showed a positive relationship between SBHCs and these outcomes. Receipt of a failing grade was examined in two studies; one found a positive relationship while the other found no impact of the SBHC intervention. Tardiness was examined in one study, which found a positive relationship between SBHC use and reduced tardiness.Seven studies examined absence and/or attendance as an outcome. Four of the studies found a positive relationship between SBHC use and fewer absences, while three of the studies found no relationship between SBHC use and absences. Three studies looked at the relationship between SBHC use and disciplinary outcomes. Two of the studies found a positive relationship between either the use and/or the presence of a SBHC with fewer disciplinary problems, while one study found no relationship. Two studies examined promotion to the next grade and two analyzed withdrawal/drop out rates; all of these studies showed a positive relationship between SBHCs and these outcomes. Receipt of a failing grade was examined in two studies; one found a positive relationship while the other found no impact of the SBHC intervention. Tardiness was examined in one study, which found a positive relationship between SBHC use and reduced tardiness.

7. Limitations of Studies on SBHC�s Impact on Academics 3 had no comparison group No �cookie cutter� SBHC model 4 examined interventions beyond the scope of typical SBHCs One used anecdotal evidence Although seven of the studies did find associations between SBHCs and selected academic indicators, most are hampered by experimental limitations. Of these seven studies, three contained no comparison group, two examined interventions that provided several services beyond the scope of typical SBHCs (Warren & i.e., after-school programs and other learning supports), two examined programs that required high levels of parental involvement (Jennings & McCord), and one was based on anecdotal evidence. Also, it is important to point out that school health services that call themselves �SBHCs� often vary enormously in the scope of their services and in their organization, so it is extremely difficult to generalize the effect, if any, that programs that called themselves �SBHCs� have on academic outcomes. At this point, therefore, we cannot conclude if there is a direct link between SBHCs and academic outcomes. Although seven of the studies did find associations between SBHCs and selected academic indicators, most are hampered by experimental limitations. Of these seven studies, three contained no comparison group, two examined interventions that provided several services beyond the scope of typical SBHCs (Warren & i.e., after-school programs and other learning supports), two examined programs that required high levels of parental involvement (Jennings & McCord), and one was based on anecdotal evidence. Also, it is important to point out that school health services that call themselves �SBHCs� often vary enormously in the scope of their services and in their organization, so it is extremely difficult to generalize the effect, if any, that programs that called themselves �SBHCs� have on academic outcomes. At this point, therefore, we cannot conclude if there is a direct link between SBHCs and academic outcomes.

8. We developed Diagram 1 to illustrate our hypothesis that three general factors might impact academic performance: student health status, social factors such as SES, and educational factors, such as class size. Of these multiple influences, the role that SBHCs could be expected to play is to improve the health status of students, which, in turn, might affect a student�s performance. We developed Diagram 1 to illustrate our hypothesis that three general factors might impact academic performance: student health status, social factors such as SES, and educational factors, such as class size. Of these multiple influences, the role that SBHCs could be expected to play is to improve the health status of students, which, in turn, might affect a student�s performance.

9. Summary of Findings Regarding Relationships Between Health Status/Risk Behaviors and Academic AchievementFactors that an SBHC might impact We thus examined seven studies that examined the relationship between health status, risk behaviors, and academic outcomes. The inclusion criteria were expanded to also include studies that examined the relationship between health status, risk behaviors, and academic outcomes but that did not focus on SBHCs specifically. These studies looked at variables that an SBHC might impact through the provision of medical, mental health, or health education services. The academic outcomes examined in these studies include WHAT? As shown in Table 2 there is a larger body of evidence documenting the impact of health-related factors (that an SBHC might impact) on academic outcomes. We thus examined seven studies that examined the relationship between health status, risk behaviors, and academic outcomes. The inclusion criteria were expanded to also include studies that examined the relationship between health status, risk behaviors, and academic outcomes but that did not focus on SBHCs specifically. These studies looked at variables that an SBHC might impact through the provision of medical, mental health, or health education services. The academic outcomes examined in these studies include WHAT? As shown in Table 2 there is a larger body of evidence documenting the impact of health-related factors (that an SBHC might impact) on academic outcomes.

10. Studies that found a Positive Relationship Between Health Status/Risk Behaviors and Academics AODT use 5 (of 5) Mental health problems 3 (of 3) Poor diet 2 (of 2) Intentional injuries 2 (of 2) Physical illness 2 (of 2) Low self-esteem and resiliency 2 (of 2) Risky sexual behaviors 1 (of 2) Health care utilization 1 (of 2) Six of the seven studies found a positive relationship with academic outcomes. These include alcohol, tobacco and other drug use (5 of 5 studies that examined this found a positive relationship), emotional problems (3 of 3), diet (2 of 2), intentional injuries (2 of 2), physical illness (2 of 2), self esteem and resilience (2 of 2), health care utilization (1 of 2) and sexual behaviors (1 of 2). Six of the seven studies found a positive relationship with academic outcomes. These include alcohol, tobacco and other drug use (5 of 5 studies that examined this found a positive relationship), emotional problems (3 of 3), diet (2 of 2), intentional injuries (2 of 2), physical illness (2 of 2), self esteem and resilience (2 of 2), health care utilization (1 of 2) and sexual behaviors (1 of 2).

11. Summary of Findings Regarding Relationships Between Health Status/Risk Behaviors and Academic AchievementFactors beyond the direct influence of an SBHC

12. Studies that found a Positive Relationship Between Other Factors and Academics Educational factors (3 of 3) Household/family characteristics (2 of 3) Demographics (2 of 2) School safety (2 of 2) Eating breakfast (1 of 1) As described in Diagram 1, it is important to note that there is a much larger body of literature on the impact of other social and educational factors that contribute to academic success or failure. This review only describes studies whose primary focus was the influence of health on academic outcomes. However, it is important to note that these studies, whose primary purpose was to examine how health impacts academics, also examined several factors that are beyond the direct influence of an SBHC. As shown in Table 3, they found that educational factors (DEFINE) (3 of 3), household/family characteristics (2 of 3), demographics (2 of 2), school safety (2 of 2) and eating breakfast (1 of 1) impacted academic outcomes impact academic outcomes. As described in Diagram 1, it is important to note that there is a much larger body of literature on the impact of other social and educational factors that contribute to academic success or failure. This review only describes studies whose primary focus was the influence of health on academic outcomes. However, it is important to note that these studies, whose primary purpose was to examine how health impacts academics, also examined several factors that are beyond the direct influence of an SBHC. As shown in Table 3, they found that educational factors (DEFINE) (3 of 3), household/family characteristics (2 of 3), demographics (2 of 2), school safety (2 of 2) and eating breakfast (1 of 1) impacted academic outcomes impact academic outcomes.

13. Conclusions from the Literature Insufficient evidence to prove a direct link between SBHCs and academic outcomes. Evidence of impact of medical and mental health status on academic outcomes. Important role of SBHCs in improving student health can contribute, at least indirectly, to improved academic outcomes. Conclusions To date, there is insufficient evidence to prove a direct link between SBHCs and academic outcomes. What has been demonstrated is the influence of medical and mental health status on academic outcomes, although there are disagreements about how much of a role health and risk behaviors actually exert as compared with other social or educational factors. Clearly, however, the role of SBHCs in improving student health can contribute, at least indirectly, to improved academic outcomes. Other literature, not reviewed in this paper, is emerging to demonstrate the importance of Coordinated School Health programs as well as other youth development and student support programs. We anticipate that the synergy of interventions including SBHCs, other health and social programs, youth development opportunities, and educational program improvements will have the highest likelihood of improving student achievement. Conclusions To date, there is insufficient evidence to prove a direct link between SBHCs and academic outcomes. What has been demonstrated is the influence of medical and mental health status on academic outcomes, although there are disagreements about how much of a role health and risk behaviors actually exert as compared with other social or educational factors. Clearly, however, the role of SBHCs in improving student health can contribute, at least indirectly, to improved academic outcomes. Other literature, not reviewed in this paper, is emerging to demonstrate the importance of Coordinated School Health programs as well as other youth development and student support programs. We anticipate that the synergy of interventions including SBHCs, other health and social programs, youth development opportunities, and educational program improvements will have the highest likelihood of improving student achievement.

14. What are Important Methodological Issues?

15. Methodological Issues Possible Outcome Measures Individual Student Data Aggregate Data Access to Data Student or Parent Survey Computerized Records Data Quality & Linking Variable Definitions

16. 1. Possible Outcome Measures Graduation rate Grade promotion GPA/grades-failing grades Credit accumulation Standardized test scores Academic Performance Index Attendance/absence Cutting class Withdrawal/drop-out rates Suspension rates Disciplinary referrals Educational aspirations

17. 2. Individual Student Data Advantage of individual student data: More power to determine differences between your control and intervention groups If individual student data is used, however, researchers may need� To obtain active parental and/or student consent Could get consent along with consent for SBHC use To comply with HIPPA issues if research study affiliated with academic institution To establish data agreements with school districts To deal with �Privacy tag issues� Many studies use aggregate data to get around issues of individual consent- talk more about implications of this later. We�ll also talk more about issues with surveys versus data from primary source. Many studies use aggregate data to get around issues of individual consent- talk more about implications of this later. We�ll also talk more about issues with surveys versus data from primary source.

18. 3. Aggregate Data Aggregate data may include students At the school or district level Who did or did not use SBHC services, and/or Who were or were not enrolled in the school for the majority of the school year Advantage of aggregate data is avoidance of consent issues If use school level data- level of analysis should be school not individual- some proceed with analyses that are inappropriate Some use individual level data that is aggregated Aggregate data -it may be appropriate to include all children in school versus users and non-users etc. depending on conceptual model of intervention- I.e. an intent to treat model might appropriately include all students and not knowing which group they fall in within the school is O.K. If use school level data- level of analysis should be school not individual- some proceed with analyses that are inappropriate Some use individual level data that is aggregated Aggregate data -it may be appropriate to include all children in school versus users and non-users etc. depending on conceptual model of intervention- I.e. an intent to treat model might appropriately include all students and not knowing which group they fall in within the school is O.K.

19. 4. Access to Data Methods of Obtaining Data Computerized or paper school records Parent/child/teacher report or survey Relationship with school information services/technology division critical Determines priority of providing data needed Level of comfort with sharing data Whether data is obtainable at all This is easier if SBHC run by school district Whenever 2 groups need to work together there are challenges in obtaining information desired in timely fashion- always easier if the data you need is something you collect and have control over- quality, process, etc. Depending on relationship with IS/IT of school you may have little control over when and what data is obtainable- providing data re: the SBHC may be low priority as it may not be seen as primary to educational mission- the better the relationship/trust- the less hesitancy there is around data sharing Sometimes no control over what data you receive- are you getting data on all students in the school or in your target group, with mobility in urban schools- what you receive may not reflect reality- some checks of this- but they may not even be able to �correct� the issue- also relates to data quality.Whenever 2 groups need to work together there are challenges in obtaining information desired in timely fashion- always easier if the data you need is something you collect and have control over- quality, process, etc. Depending on relationship with IS/IT of school you may have little control over when and what data is obtainable- providing data re: the SBHC may be low priority as it may not be seen as primary to educational mission- the better the relationship/trust- the less hesitancy there is around data sharing Sometimes no control over what data you receive- are you getting data on all students in the school or in your target group, with mobility in urban schools- what you receive may not reflect reality- some checks of this- but they may not even be able to �correct� the issue- also relates to data quality.

20. 5. Student or Parent Surveys Limitations of surveys Recall bias How questions are phrased Low response rates Non-representative sample How question is asked: For example: absences, an exact number is easier to remember if your child has few absences, but harder the more frequently absent a child is- accuracy More of an issue if variable varies widely within a school being studied or between schools being studied How question is asked: For example: absences, an exact number is easier to remember if your child has few absences, but harder the more frequently absent a child is- accuracy More of an issue if variable varies widely within a school being studied or between schools being studied

21. 6. Computerized Records May be more accurate than surveys, but still some issues Where student mobility is high, absences in particular may be inaccurate- absences may be high due to lack of knowledge that student has moved and has not officially withdrawn Difficulty with decentralized school districts May not be designed to collect all variables of interest Decentralized school district- data entered at individual school level- accuracy improved by district policies and training on data entry - however still variation motivations may be different at different schools to record items in different ways, maintain students as enrolled or not enrolled based on financial incentives associated with that- some places finances geared at total enrollment- others based on students and absences�not within district- but especially between district evaluation this importantDecentralized school district- data entered at individual school level- accuracy improved by district policies and training on data entry - however still variation motivations may be different at different schools to record items in different ways, maintain students as enrolled or not enrolled based on financial incentives associated with that- some places finances geared at total enrollment- others based on students and absences�not within district- but especially between district evaluation this important

22. 7. Data Quality & Linking Data Quality Limits on how data can be verified Limited by cost Limited by how data provided Data Linking Different sources, different challenges If multiple sources, how will data be linked Consent issues Use of unique identifiers Making data non-identifiable before link Data agreements that allow links by certain personnel For example you get a computerized data file with individual student records- with enrollment history and absences at end of school year- some children you note have over 100 days absent- at this point in time the child may not be in the school or district and you can�t verify. You could set up small validation study- where for 2-4 weeks you have teachers record absences of certain students and compared to computerized record- time consuming/practical? Brings up issue of program evaluations and research- the �standard� may be different- but conclusions made by the �audience� at large when this information is shared might not distinguish the 2- how this impacts our ability to make sound conclusions about academic outcomes.For example you get a computerized data file with individual student records- with enrollment history and absences at end of school year- some children you note have over 100 days absent- at this point in time the child may not be in the school or district and you can�t verify. You could set up small validation study- where for 2-4 weeks you have teachers record absences of certain students and compared to computerized record- time consuming/practical? Brings up issue of program evaluations and research- the �standard� may be different- but conclusions made by the �audience� at large when this information is shared might not distinguish the 2- how this impacts our ability to make sound conclusions about academic outcomes.

23. 8. Variable Definitions Caution to make sure there are uniform definitions for variables being measured Districts using different criteria for determining and coding whether a child is exempt from academic testing Schools within a district using different criteria to determine whether an absence is excused/unexcused or what constitutes being absent ( full day/half day)

24. Given what we know, what are the next questions to ask about the relationship between SBHCs and academic outcomes?

25. Keep in mind�. We know: SBHCs improve access to health care Medical and mental health status of students impacts their academic success And: All SBHCs are not exactly the same but are defined by their common elements SBHCs are a part of a variety of services offered in schools

26. However� The relationships between SBHCs and academic outcomes that we postulate as research and evaluation questions MAY be direct or indirect.

27. And this relationship should be� Possible (SBHC actually provides the intervention that is expected to influence academic outcome) Realistic (takes into account multiple interventions that may occur simultaneously or that may all impact academic outcome) Logical (model linking the intervention with the outcome can be described) Demonstrable (data for the intervention and outcome or adequate proxies can be reasonably obtained)

28. Where should we go next? Consider the various levels at which the intervention can be defined. Is it� The SBHC as a whole or A specified array of services delivered in the SBHC context or An intervention delivered to a specific sub-population in the SBHC?

29. And� Examine the findings when each of these definitions are used. More focused interventions may yield clearer links to academic outcomes. Consider how SBHCs contribute to the Coordinated School Health program model as well as youth development and student support programs.

30. Finally!! Consider that the highest likelihood of improving student achievement may come from the synergy of interventions including SBHCs, other health and social programs, youth development opportunities, and educational program improvements. Develop a variety of algorithms or models that describe the relationships that we think occur based on our prior research and empirical observations.

31. Health Risk Behaviors: Educational Behaviors: Educational Outcomes: Health Risk Behaviors: Educational Behaviors: Educational Outcomes: