Susan L. Tarasevich, NE RET Associate Carl Fertman, NE RET Associate

1. Developing a Student Assistance Program Logic Modelfor the Rhode Island Department of Mental Health, Retardation and Hospitals, and the Rhode Island Student Assistance Provider AgenciesJune 21, 2010 Susan L. Tarasevich, NE RET Associate Carl Fertman, NE RET Associate Chuck Klevgaard, NE RET Coordinator

2. Agenda • Review of Logic Model templates and Project SUCCESS (PS) Logic Model • Create RI SAP Logic Model to guide program planning and evaluation • Considering theory base: possible instrumentation  • Outcome and impact measures; evaluation questions • Describe measures of fidelity to the PS Model 

3. Components of RI SAP Prevention ____________________________ ____________________________ ____________________________ ____________________________ ____________________________ ____________________________

4. Components of RI SAP Intervention ____________________________ ____________________________ ____________________________ ____________________________ ____________________________ ____________________________

5. Components of RI SAP Support & Continuing Care ____________________________ ____________________________ ____________________________ ____________________________ ____________________________ ____________________________

6. A logic model is… • A depiction of a program showing what the program will do and what it is to accomplish • A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes • The core of program planning and evaluation University of Wisconsin-Extension, Program Development and Evaluation

7. Simplest form INPUTS OUTPUTS OUTCOMES University of Wisconsin-Extension, Program Development and Evaluation

8. What a logic model is not… • A theory • Reality • An evaluation model or method It is a framework for describing the relationships between investments, activities, and results. It provides a common approach for integrating planning, implementation, evaluation and reporting. University of Wisconsin-Extension, Program Development and Evaluation

9. Logic modeling is a way of thinking…not just a pretty graphic “We build the road and the road builds us.” -Sri Lankan saying University of Wisconsin-Extension, Program Development and Evaluation

10. OUTPUTS OUTCOMES INPUTS Program investments Activities Short Participation Medium Long-Term What we do Who we reach What we invest What results A logical chain of connections showing what the program is to accomplish University of Wisconsin-Extension, Program Development and Evaluation

11. If-Then Relationships Underlying a logic model is a series of ‘if-then’ relationships that express the program’s theory of change University of Wisconsin-Extension, Program Development and Evaluation

12. PC’s Theoretical Constructs Project SUCCESS delineates four possible constructs for change: • IOM Prevention Framework • Risk/Protective Factors & Resiliency • Perceived Social Support • Increased Parental Involvement

13. Six Steps in Logic Model Development Step One: Describe the Need for SAP Step Two: Defining Goals of RI SAP Step Three: Identify the Inputs Step Four: Outline the Program Activities Step Five: Identify Process Measures Step Six: Identify Outcome – Impact Measures

14. Step One: Describe the Need • What problem exists that demands a response? • Why does it exist? • For whom does it exist? • Who has a stake in the problem? • What can be changed? • Traps: • Framing a problem as a need where need is actually a program or service.

15. The Burden of the Problem Studies have shown the annual cost of substance abuse to the Nation to be $510.8 billion in 1999 (Harwood, 2000). • Alcohol abuse cost the Nation $191.6 billion; • Tobacco use cost the Nation $167.8 billion; • Drug abuse cost the Nation $151.4 billion.

16. The Burden of the Problem Substance abuse clearly is among the most costly health problems in the United States. • Among national estimates of the costs of illness for 33 diseases and conditions, alcohol ranked second, tobacco ranked sixth, and drug disorders ranked seventh (National Institutes of Health [NIH], 2000). • This report shows that programs designed to prevent substance abuse can reduce these costs.

17. The Burden of the Problem Although 80 percent of American youth reported participation in school-based prevention in 2005 (SAMHSA, 2004), only 20 percent were exposed to effective prevention programs (Flewelling et al., 2005). • Given this level of participation, it is possible that some expected benefits already exist for these students, and the estimates in this paper are adjusted for these probable benefits. These cost-benefit estimates show that effective school-based programs could save $18 for every $1 spent on these programs.

18. Rhode Island v. United States • The Youth Risk Behavior Survey, 2009 report • Youth Online: CDC-Youth Online-High School YRBS T-Test Rhode Island 2009 and United States 2009 Results Need internet access to connect

19. Example: Need Statements • Local data indicate that the substance abuse among young adults aged 14 to 18, while better than national averages, continues to be a major health concern • SAPs have been implemented for over a decade across the United States, but few have been rigorously evaluated • SAP is not a program; rather it is a set of practices based on multiple theoretical constructs

20. Step Two: Defining the Goals • Goal statements describe the intended long-term outcomes of your initiative in large, broad terms. • Examples of relevant goals for the PS (Morehouse) initiative are: • Preventing and reducing substance abuse in adolescents ages 14-18 • Reducing problem behaviors • Decreased association with using peers

21. Goals • A goal represents a general, big-picture statement of desired results. Form as a question. • ‘What are issues that you would like the program to address?’ (e.g., the goal of the program is to reduce and delay substance abuse in adolescents ages 14 to 18) University of Wisconsin-Extension, Program Development and Evaluation

22. Outcomes Answers the question: ‘What changes do you want to occur because of your program?’ (e.g., the outcome of the program will be to increase the number of students showing increases in social skill development, resiliency, increase perceived social support)

23. Step Three: Identify the Inputs • Resources utilized to operate the program personnel, fiscal and other resources - List the resources • Who has a vested interest in how the program is delivered and its results ? - List the stakeholders

24. Step Four: Outline the Activities • Activities are the things that you do to accomplish your goals • Informed by your needs, goals • Based on a theory or theories of change • Examples of some activities for the PS initiative include: • Referral to SAC • School-Based Assessment & Action Plan • Referral to Selective Skill Development Group • Referral for Treatment

25. Step Five: Identify Process Measures Process measures or outputs are the instruments or tools used to capture data on the implementation of activities (e.g., attendance sheets; registration forms; program fidelity or observation checklists). Process data • Implementation of activities • Participants served (e.g., number of people served, number of sessions delivered, number of staff trained).

26. Step Six: Identify Outcome-Impact Measures Outcome measures or indicators are the source of evidence for determining whether or not your program is reaching its specified objectives. • Short-Term: One to Three Years • Changes in knowledge, skills & attitudes • Intermediate: Three to Five Years • Changes in behavior, norms, and/or policies • Long-Term: Four to Six Years • changes in organizations and systems

27. C H A I N OF O U T C O M E S University of Wisconsin-Extension, Program Development and Evaluation

28. Considerations • Identify: • what will change for whom • by how much • by when • Typically related to changes in knowledge, attitudes, skills, and behaviors

29. Inputs Outputs Outcomes Activities Participation Short Medium Long-term Inputs Activities Outputs Outcomes Long-term Medium Common Variations UWEX Logic Model Other common logic model used by United Way, Center for Disease Control and others Short University of Wisconsin-Extension, Program Development and Evaluation

30. Check Your Logic Model • Is it meaningful? • Does it make sense? • Is it doable? • Can it be verified? University of Wisconsin-Extension, Program Development and Evaluation

31. INPUTS OUTCOMES OUTPUTS Match Evaluation Questions to Program Medium Activities Participation Short Long-Term Program Investment Evaluation questions:What questions do you want to answer?E.g., accomplishments at each step; expected causal links; unintended consequences or chains of events set into motion Indicators: What evidence do you need to answer your questions? University of Wisconsin-Extension, Program Development and Evaluation

32. Data Collection Plan University of Wisconsin-Extension, Program Development and Evaluation

33. Resources Miller, T. and Hendrie, D. (2009). Substance Abuse Prevention Dollars and Cents: A Cost-Benefit Analysis, DHHS Pub. No. (SMA) 07-4298. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration. Youth Risk Behavior Survey System (2010). Atlanta, GA. Centers for Disease Control. Downloaded from the Web: http://www.cdc.gov/healthyyouth/yrbs/index.htm

34. Resources Miller, T. and Hendrie, D. (2009). Substance Abuse Prevention Dollars and Cents: A Cost-Benefit Analysis, DHHS Pub. No. (SMA) 07-4298. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration. Youth Risk Behavior Survey System (2010). Atlanta, GA. Centers for Disease Control. Downloaded from the Web: http://www.cdc.gov/healthyyouth/yrbs/index.htm

35. Presenters Carl I. Fertman PhD, MBA, CHES Associate Professor University of Pittsburgh 155 Trees Hall Pittsburgh, PA 15261 412.648.7191 (Office) 412.648.7198 (Fax) Carl@pitt.edu Chuck Klevgaard, BSW Regional Coordinator, CSAP’s CAPT Northeast Regional Expert Team (NE RET) Education Development Center 55 Chapel Street, Newton, MA 02458 617.969.7100 (Office) 617.244.3436 (Fax) CKlevgaard@edc.org Susan L. Tarasevich, Ed.D. Tarasevich Consults 1330 Breezewood Drive West Homestead, PA. 15120 412.848.8055 (Cell Phone) 412.586.2891 (Fax) tarasl@upmc.edu