IMPLEMENTING COMPLEX EVIDENCE-BASED PROGRAMS

1. IMPLEMENTING COMPLEX EVIDENCE-BASED PROGRAMS FROM SERVICE TO SCIENCE AND BACK AGAIN REAP Conference Santa Fe, NM March 20, 2008

2. RESEARCH TEAM Susan Carter, PI susancarter@hughes.net Gladys Levis-Pilz cowalden@yahoo.com Heather Yazzie heatheryazzie@yahoo.com Support provided by a FIRE grant from the Office of Juvenile Justice and Delinquency Prevention, US DOJ STUDY DETAILS

3. THE PROBLEM “We are faced with the paradox of non-evidence-based implementation of evidence-based programs” (Drake, Gorman, & Torrey, 2002) Great emphasis on identifying “evidence-based” programs, but little attention paid to their implementation beyond “fidelity” discussions, especially for complex programs

4. RESEARCH QUESTIONS(After Fixen et al, NIRN, 2005) BASED ON STAGES OF PROGRAM IMPLEMENTATION • Exploration and Adoption • Program Installation • Initial Implementation • Full Operation • Innovation and Sustainability

5. DESIGN AND METHODS BASED ON NIYLP’S PROJECT VENTURE • Created by McClellan Hall, Founder of National Indian Youth Leadership Project, Gallup, NM • Grew out of camp program into year round model in 1990 • Named by NREPP as a Model Program in 2004 • Re-reviewed by NREPP, designated “evidence-based” program in 2007 • “Universal” prevention model based on experiential, youth development principles for Native American mid-school youth

6. Project Venture 3 levels of “non-negotiables” Populationserved Programcomponents Program delivery • Focus on Native youth • Middle school aged youth* • High school aged Service Staff • Mix of risk levels (program should not have stigma of “prevention program”)* • Positive approach • Outdoor adventure and experiential challenge • Experiential education • Wilderness outdoor adventure • Physical education • Nature challenge • Service learning • Active learning • Academics in action • School or community service • Parent involvement • Native culture based leadership • Culture and values • Language and history • Cross-cultural exchange and awareness • Elders • 200 hours of contact annually • Risk management principles followed, including sequential progression of elements including • Physical safety • Mental health safety • Youth to follow four levels of progression through program • In school (highly recommended) • After school • Weekend and overnight • Intensive camp experience • Outcomes measured Bold: Basic non-negotiable (element must be in place, even from beginning) Regular: Phased-in non-negotiable (element must be in place, but may be phased in over time) Italic: Recommendation (element is recommended, but not required) *: Modification allowed (element may differ if program is appropriately modified)

7. DESIGN AND METHODS • Phase 1: In-depth Case Studies of a few PV programs • Identify all known PV adoption sites in US • Select 8 “diverse” sites considering rural/urban, new/experienced agency, agency type, etc. • Send invitation letter • Conduct on-site program observations • Conduct semi-structured interviews • Review documents related to implementation • Complete PV Replication Instrument • Complete “saw/read/heard about it” table

8. Project Venture: currently ~36 replication sites = PV replication site There are approximately 36 replication sites across 17 states

9. Exploration and Adoption How did you hear about PV in the first place? How/why was the decision made to select? What are the components of your program? Is the programming running fully or partially? Community Support? School Support? Who makes the decisions for PV? Is it important that your facilitators be American Indian? Program Installation How did you prepare to implement PV? Who made those decisions? Staffing? Financing? Restructuring? What about now, how is PV implemented? Staffing? Financing? Restructuring? Initial Implementation Participants, who? How are they selected? Risk levels? PV components: which?, how configured? Full Operation How did you implement PV components? Did you begin with all components or phase them in? Who benefits? How do you feel about PV? What kinds of out comes have you had? Are you measuring outcomes? Tell me about this. Would you share data with NIYLP? Do you use it to improve programs? Does your program hold true to the vision of PV? (go through Program Implementation instrument with them) Innovation and Sustainability How do you manage change in population, financing, community political/tribal, drift/ fidelity? Any plans to sustain? How? Interview Guide Questions

11. DESIGN AND METHODS • Phase 2: Broad study of all PV programs • Develop survey based on Phase 1 findings • Administer via web, mail, phone to all known PV sites

12. PRELIMINARY FINDINGS FROM CASE STUDIES From earlier Project Venture studies, Bloom’s Taxonomy seemed a useful framework upon which to hang observational data “Life lessons” available to individual youth increased through exposure to other facilitators and participants and these lessons also promoted higher levels of cognitive and affective learning (as detailed below).

13. CASE STUDY FINDINGS (CONT.) The researcher’s attention kept returning to an idea of “evolution” or levels resulting from provider-participant and participant-participant interactions. These thoughts in turn led to a review of the educational psychologist Benjamin Bloom’s taxonomy of “levels of learning” (Bloom, B. (ed.) Taxonomy of Educational Objectives: The Classification of Educational Goals,1956, New York: Longmans Green).

14. CASE STUDY FINDINGS (CONT.) This classic theory provides a classification of levels of thinking and feeling behaviors thought to be important in the process of learning. The taxonomies are both descriptive and prescriptive as it orders categories of cognitive, affective, and psychomotor skills. From basic/”low” levels to ever more abstract/”higher” levels of learning.

15. While experimental studies are crucial for advancing our understanding of what works in adventure programming, the qualitative perspective has the power to show us how programming works. Intentionally designed and executed qualitative research enriches our understanding of what goes on in the black box between pre and post tests and provides us with the stories to tell and prescriptions for action.

16. Bloom’s Taxonomy: A Promising Analytic Framework Cognitive Skills (from less to more complex) • Knowledge • Comprehension • Application • Analysis • Synthesis • Evaluation

17. Bloom’s (cont.) Affective Skills (from less to more complex) • Receiving phenomena • Responding to phenomena • Valuing • Organization • Internalizing values