Evidence Based Programs

1. Evidence Based Programs Dr. Carol Albrecht Utah State Extension Assessment Specialist carol.albrecht@usu.edu

2. Evidence Based Programs • Relatively new approach • Program decisions are based primarily on scientific method and scientific evidence • Use best evidence available to make decisions about program • Recognizes that knowledge is provisional and subject to refutation

3. Evidence Based Programs combines (1) Program Manager’s Expertise, (2) Client Attributes, and (3) Research Findings EBP

4. Best Research Evidence Available Can Be Found in Current Literature Step 1 – Review The Literature • “Track down” the evidence- ongoing process • Note Inconsistencies • Note research suggesting some interventions are ineffective Step 2 – Critique the Literature You Find • Examine methods used*

5. Client Attributes • Take values, concerns, expectations of client into consideration • Involve clients in decision making process • Determine if this program will “fits” your client. • Not “one size fits all” situation • Consider race, gender, idiosyncratic characteristics

6. Program Manager Expertise • Use Common Sense • Avoid “fallacies” which include • Overgeneralization - Stereotyping • Selective Perception/Observation • Ex-post Facto Hypothesizing • Provide alternative explanation after the “study” • Accepting “new” as better • Accepting “traditional” as better

7. How to Complete a Literature Review • Step 1 – Formulate a Question • Step 2 – Search for Evidence • Step 3 – Critically Appraise Relevant Studies • Step 4 – Apply the Literature to Your Question and/or Program • Step 5 – Conduct Your own Research

8. Step 1 – Formulate a Question • What objectives do you want to accomplish? • What intervention is necessary to reach objectives? • Does Intervention A work? • Is Intervention A cost effective? • Does Intervention A work better than Intervention B? • Does Intervention A work for this type of client? • Does intervention A work in this kind of situation?

9. Useful Acronym for Formulating Question • Use acronym CIAO • Client characteristics • Intervention being considered • Alternative intervention being considered (if any) • Outcome of intervention (does it work?)

10. Example • Client Characteristics If children involved in 4-H • Intervention …..personally interact with a mentor for several years….. • Alternative Intervention …..or communicate with a mentor over the internet….. • Outcome …..are they more likely to seek post-high school education?

11. Step 2 – Search for the Evidence • Helpful “hints” to finding articles • Use Google or Google Scholar and type in correct search terms • Combine general terms (e.g., volunteers, evaluation etc.) • With more specific terms (i.e., grades in high school, crop yield etc.) • Top Down and Bottom Up Searches (Mullen 2006) • Top Down – Find Articles/Books that have Compiled the Literature (i.e., meta-analysis on your topic) • Bottom Up – Find a very recently published article and look at references.

12. Examples of Resources for “Top Down” Searches • *Campbell collaboration: http://www.campbellcollaboration.org/index.html • **Cochrane Collaboration: http://www.cochrane.org • Substance Abuse: http://nrepp.samhsa.gov/htpp:/www.modelprograms.samhsa.gov/index. htm • Crisis Intervention and Domestic Violence National Resource: • http://www.crisisinterventionnetwork.com • Expert Consensus Guidelines series: http://www.psychguides.com • National Guidelines Clearinghouse http://www.guidelines.gov • National Institute on Drug Abuse: http://www.nida.nih.gov/ • Substance Abuse and Mental Health Services Administration: http://www.samhsa.govindex.aspx *Focuses on Social Welfare, Education and Criminal Justice **Focuses on Health Care Interventions

13. Step 3 – Critically Appraise Relevant Articles • Emphasis Should be on Methods Used to Collect Empirical Information

14. How to Complete a Literature Review • Step 1 – Formulate a Question • Step 2 – Search for Evidence • Step 3 – Critically Appraise Relevant Studies • Step 4 – Apply the Literature to Your Question and/or Program • Step 5 – Conduct Your own Research

15. Step 4 – Apply the Literature to Your Research Question • Which intervention currently has the best empirical support? • Is the situation same/similar to yours? • Are the participants in the study the same or similar to the ones in your program/study?

16. Step 5 – Conduct Your Own Research • Use Literature Review to help you develop valid and reliable indicators. • Use Literature Review to help you determine your sampling technique and sample size. • Use the Literature Review to help you determine the most appropriate type of data collection. • Use the Literature Review to help you determine ways you can further contribute to the research (i.e., what is missing?) • Evaluate impact of program on your participants

17. Concerns About Evidence Based Programs • Merely Cost Cutting • Have to use the methods “in stock” • Evidence Based May not be Applicable • Leads to rigid implementation • Your participants may be different • Evidence is in Short Supply – not enough research • Real World Obstacles Hinder Implementation of Programs • Time and money are scarce • Devalues Qualitative Research • Argument that “Empirical Evidence” is not obtainable

18. Suggestions for Appraising Journal Articles in the Literature • Step 1 – Identify Purpose • Stated purpose of article • This should appear in first paragraph(s). • Step 2 – Literature Review • Is it adequate? • They should have at least 15 references.

19. Appraising Literature • Step 3 - Identify objectives/outcomes • Do they identify and define their outcomes/objecties • It must be clear and succinct. • They must be clearly related to the implementation of program. • How do they measure their concepts/outcomes? • Are measurements valid? (face validity) • Are measurement reliable (consistent) • Could you repeat study?

20. Appraising Literature • Step 4 – Appraising Sampling Technique • It must be clearly specified? • What is their unit of analysis? • Do you know exactly how these units will be selected? • It must be appropriate? • Does the sampling technique “fit” with the data collection technique? • Do they draw valid conclusions based on sampling technique? • Note – if they generalize they must have random sample

21. Remember the Sampling Techniques • Random Sample – can generalize • Simple or Systematic Random • Multi-stage Cluster • Stratified Random • NonRandom Sample – cannot generalize • Theoretical or Purposive • Snowball • Convenient

22. Appraising Literature • Step 5 – Research Design/Data Collection Technique • It must be clearly specified? • Do you know exactly how they will collect their data? • It must be appropriate • Keep in mind the strengths and weaknesses of the different research designs/data collection techniques

23. Remember Research Designs or Data Collection Techniques • Data Collection/Research Design • Survey or Questionnaire • Secondary • Experimental • Lab Experiments • Quasi – Field Experiment/Trial • Field Research (Qualitative) • Participant Observation • Content Analysis • Intensive Interviews • Case Studies

24. Appraising Literature • Step 6 - Critique Tables/Graphs/Charts • Their analysis must be correct? • Did they include descriptive statistics of the sample? • Did they include inferential statistics and discuss level of significance? • Are their tables constructed correctly? • They must be interpreted accurately. • Does the text accurately describe the data? • They should look professional • Do they provide sufficient information? • Do they tell you how they measured the indicators used in the tables?

25. Appraising Literature • Step 7 – Summary and Conclusions • Do they accurately summarize their research? • Do they identify strengths and weaknesses? • Do they acknowledge limitations? • Do they make suggestions for future research?

26. Easy Steps • 1. Identify Purpose • 2. Critique Literature Review • 3. Identify Success • 4. Sample Technique • 5. Data Collection Technique/Research Design • 6. Tables and Charts • 7. Summary and Conclusions

27. Contact Information • Dr. Carol Albrecht • Assessment Specialist • Utah State University • 979-777-2421 • Carol.albrecht@usu.edu